Nov 03 05:54:30.785091 ip-10-0-132-233 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 05:54:31.223551 ip-10-0-132-233 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:54:31.223551 ip-10-0-132-233 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 05:54:31.223551 ip-10-0-132-233 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:54:31.223551 ip-10-0-132-233 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:54:31.227029 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.226936    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.227029 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227028    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.227029 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227031    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.227029 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227034    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227037    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227040    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227043    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227046    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227048    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227052    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227055    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227058    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227061    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227064    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227066    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227069    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227071    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227074    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227081    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227085    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227087    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227090    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227093    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.227146 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227096    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227099    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227101    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227104    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227106    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227109    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227112    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227114    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227117    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227119    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227122    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227124    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227127    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227130    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227133    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227136    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227139    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227141    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227144    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.227633 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227146    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227149    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227152    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227155    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227157    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227160    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227163    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227166    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227169    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227172    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227174    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227177    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227180    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227188    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227193    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227197    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227199    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227202    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227204    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227208    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.228098 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227211    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227214    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227216    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227219    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227221    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227224    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227227    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227231    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227233    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227254    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227257    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227259    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227262    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227265    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227268    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227271    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227275    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227279    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227282    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.228574 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227284    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227287    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227289    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227292    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227295    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227298    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227301    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227303    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227306    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227308    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227311    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227313    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227316    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227319    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227322    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227324    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227328    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227332    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227335    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.229037 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227339    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227342    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227346    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227349    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227351    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.227354    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228183    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228191    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228194    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228197    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228200    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228203    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228206    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228209    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228211    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228214    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228217    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228221    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228223    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228226    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.229504 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228229    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228232    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228234    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228237    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228239    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228242    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228245    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228247    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228250    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228253    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228256    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228259    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228262    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228265    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228268    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228271    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228274    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228277    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228279    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.229981 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228282    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228285    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228287    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228290    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228293    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228295    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228298    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228301    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228304    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228306    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228309    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228312    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228315    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228317    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228320    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228323    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228325    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228328    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228330    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228333    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.230443 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228335    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228338    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228340    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228343    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228346    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228348    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228352    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228354    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228357    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228360    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228362    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228365    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228367    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228370    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228373    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228375    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228378    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228381    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228383    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.230928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228386    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228388    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228391    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228393    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228396    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228399    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228402    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228405    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228408    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228410    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228413    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228416    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228418    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228421    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228423    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228426    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228429    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228432    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228435    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228438    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.231411 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228440    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228443    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228446    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228461    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228464    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228466    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228469    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228471    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228475    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228479    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228482    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228487    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228492    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.228495    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229004    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229011    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229014    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229018    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229021    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.231928 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229023    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229026    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229029    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229032    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229034    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229037    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229040    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229043    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229046    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229049    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229052    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229054    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229057    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229060    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229062    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229065    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229067    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229070    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229072    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229075    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.232642 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229078    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229080    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229082    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229085    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229088    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229090    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229093    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229095    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229099    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229103    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229106    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229110    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229112    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229115    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229119    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229122    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229124    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229127    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.233136 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229129    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229132    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229135    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229138    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229140    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229143    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229146    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229149    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229151    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229154    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229157    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229159    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229161    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229164    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229166    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229169    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229171    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229174    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229176    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229179    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.233545 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229182    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229184    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229187    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229189    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229192    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229194    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229197    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229200    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229202    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229205    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229207    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229210    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229212    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229215    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229218    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229220    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229223    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229226    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229229    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229232    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.234013 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229234    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229237    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229239    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229242    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229246    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229248    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229251    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229253    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229256    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229258    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229261    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229263    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229266    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229268    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229271    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229273    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229276    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229279    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229281    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229286    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.234512 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229288    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229291    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.229293    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230329    2582 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230340    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230347    2582 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230353    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230358    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230361    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230367    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230372    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230375    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230379    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230382    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230386    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230389    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230392    2582 flags.go:64] FLAG: --cgroup-root=""
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230395    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230398    2582 flags.go:64] FLAG: --client-ca-file=""
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230401    2582 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230404    2582 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230410    2582 flags.go:64] FLAG: --cluster-domain=""
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230413    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 05:54:31.234978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230416    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230419    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230422    2582 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230427    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230430    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230436    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230439    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230442    2582 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230446    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230450    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230453    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230456    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230461    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230464    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230468    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230470    2582 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230474    2582 flags.go:64] FLAG: --enable-server="true"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230478    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230483    2582 flags.go:64] FLAG: --event-burst="100"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230486    2582 flags.go:64] FLAG: --event-qps="50"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230489    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230492    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230495    2582 flags.go:64] FLAG: --eviction-hard=""
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230500    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 05:54:31.235535 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230503    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230507    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230510    2582 flags.go:64] FLAG: --eviction-soft=""
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230513    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230516    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230519    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230522    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230525    2582 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230528    2582 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230531    2582 flags.go:64] FLAG: --feature-gates=""
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230539    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230542    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230545    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230552    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230555    2582 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230558    2582 flags.go:64] FLAG: --help="false"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230561    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230565    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230568    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230571    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230574    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230578    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230581    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 05:54:31.236138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230583    2582 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230586    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230589    2582 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230593    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230596    2582 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230599    2582 flags.go:64] FLAG: --kube-reserved=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230602    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230605    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230608    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230611    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230614    2582 flags.go:64] FLAG: --lock-file=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230616    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230620    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230623    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230629    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230632    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230635    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230638    2582 flags.go:64] FLAG: --logging-format="text"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230641    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230644    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230647    2582 flags.go:64] FLAG: --manifest-url=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230650    2582 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230656    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230660    2582 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230664    2582 flags.go:64] FLAG: --max-pods="110"
Nov 03 05:54:31.236674 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230670    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230673    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230676    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230679    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230683    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230685    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230688    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230696    2582 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230700    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230703    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230708    2582 flags.go:64] FLAG: --pod-cidr=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230711    2582 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230714    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230717    2582 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230720    2582 flags.go:64] FLAG: --port="10250"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230723    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230726    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf09f0733869f80a"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230729    2582 flags.go:64] FLAG: --qos-reserved=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230732    2582 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230735    2582 flags.go:64] FLAG: --register-node="true"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230738    2582 flags.go:64] FLAG: --register-with-taints=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230743    2582 flags.go:64] FLAG: --registry-burst="10"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230745    2582 flags.go:64] FLAG: --registry-qps="5"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230748    2582 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230751    2582 flags.go:64] FLAG: --reserved-memory=""
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230755    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 05:54:31.237252 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230758    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230761    2582 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230764    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230768    2582 flags.go:64] FLAG: --runonce="false"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230771    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230775    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230777    2582 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230780    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230783    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230787    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230800    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230804    2582 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230807    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230810    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230813    2582 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230816    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230821    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230824    2582 flags.go:64] FLAG: --system-cgroups=""
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230827    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230833    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230836    2582 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230839    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230844    2582 flags.go:64] FLAG: --tls-min-version=""
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230847    2582 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230849    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 05:54:31.237905 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230852    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230855    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230859    2582 flags.go:64] FLAG: --v="2"
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230863    2582 flags.go:64] FLAG: --version="false"
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230868    2582 flags.go:64] FLAG: --vmodule=""
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230872    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.230876    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231012    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231016    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231019    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231022    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231025    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231028    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231030    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231033    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231036    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231040    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231044    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231047    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231072    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231076    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.238473 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231080    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231083    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231087    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231090    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231092    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231095    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231098    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231100    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231103    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231105    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231108    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231110    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231113    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231116    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231119    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231121    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231124    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231127    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231129    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231131    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.238966 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231134    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231137    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231140    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231143    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231145    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231148    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231150    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231153    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231157    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231160    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231164    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231168    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231171    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231173    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231178    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231180    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231192    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231196    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231199    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.239459 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231201    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231204    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231206    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231209    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231211    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231214    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231216    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231237    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231241    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231243    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231246    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231248    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231251    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231254    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231257    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231260    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231262    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231265    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231267    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.239931 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231270    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231272    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231275    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231278    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231280    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231283    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231286    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231289    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231293    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231296    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231299    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231301    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231304    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231306    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231309    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231311    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231315    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231317    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231320    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231323    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.240389 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231325    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231328    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231331    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231333    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231336    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231338    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231341    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231344    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231346    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231349    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231352    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231354    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231357    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231359    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.231389    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231430    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.240880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231436    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231439    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231442    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231445    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231450    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231452    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231455    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231458    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231461    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231463    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231466    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231469    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231471    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231474    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231477    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231479    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231482    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231484    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231487    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.241239 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231489    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231492    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231494    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231497    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231501    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231504    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231507    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231510    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231512    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231515    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231518    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231521    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231523    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231526    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231529    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231531    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231534    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231539    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231541    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231544    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.241675 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231547    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231549    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231552    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231555    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231557    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231560    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231562    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231565    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231568    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231570    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231573    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231575    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231578    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231581    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231583    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231586    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231590    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231593    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231595    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.242301 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231598    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231601    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231603    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231606    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231610    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231614    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231618    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231622    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231625    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231629    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231633    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231636    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231639    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231641    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231644    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231647    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231649    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231652    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231654    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231657    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.242828 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231659    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231662    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231665    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231667    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231670    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231673    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231675    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231678    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231680    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231684    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231687    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231690    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231693    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231695    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231698    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231701    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231703    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231707    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231711    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231714    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.243345 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231718    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231721    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231727    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.231804    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.233607    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.233626    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.233632    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.241048    2582 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.241068    2582 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.241871    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.241879    2582 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241948    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241953    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241956    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241960    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241963    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241965    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241968    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241970    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.243838 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241973    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241976    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241978    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241981    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241984    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241987    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241992    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.241999    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242003    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242005    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242008    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242011    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242015    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242020    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242023    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242026    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242029    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242031    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242035    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.244297 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242037    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242040    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242043    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242046    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242048    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242051    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242053    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242056    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242059    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242061    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242064    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242067    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242069    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242072    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242075    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242078    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242080    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242083    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242086    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.244738 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242090    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242092    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242095    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242098    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242100    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242103    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242105    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242108    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242110    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242113    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242115    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242118    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242121    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242123    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242126    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242128    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242131    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242134    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242136    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242139    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.245171 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242141    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242144    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242147    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242149    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242151    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242154    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242156    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242159    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242162    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242165    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242168    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242171    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242173    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242176    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242178    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242181    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242184    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242186    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242189    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242192    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.245635 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242195    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242197    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242199    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242202    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242205    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242207    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242210    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242213    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242215    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242218    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242220    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242223    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242226    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242228    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242231    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242233    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242236    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242238    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242241    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.246155 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242243    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.242277    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242439    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242444    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242448    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242451    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242454    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242459    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242462    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242465    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242468    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242470    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242473    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242475    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242478    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.246600 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242481    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242484    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242487    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242489    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242492    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242495    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242497    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242500    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242503    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242505    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242508    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242510    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242513    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242515    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242518    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242520    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242523    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242525    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242529    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242531    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.246949 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242534    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242537    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242540    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242543    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242545    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242548    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242550    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242553    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242557    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242560    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242563    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242566    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242568    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242571    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242573    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242576    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242579    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242581    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242583    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.247441 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242586    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242589    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242591    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242593    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242596    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242599    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242601    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242604    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242607    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242609    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242612    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242615    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242617    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242620    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242622    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242625    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242628    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242630    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242633    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.247900 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242636    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242638    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242641    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242643    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242646    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242648    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242651    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242656    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242658    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242661    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242663    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242666    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242668    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242671    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242674    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242676    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242679    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242681    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242684    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242686    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.248372 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242689    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242692    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242694    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242697    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242700    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242703    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242705    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242708    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242711    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242713    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242716    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242719    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242721    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242725    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242727    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.242750    2582 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:54:31.248843 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242782    2582 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242786    2582 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242804    2582 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242809    2582 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242812    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242816    2582 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242820    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242823    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242826    2582 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242829    2582 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242831    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242834    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242840    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242843    2582 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242845    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242848    2582 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242850    2582 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242853    2582 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242856    2582 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:54:31.249218 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242858    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242861    2582 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242864    2582 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242867    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242869    2582 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242872    2582 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242874    2582 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242877    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242880    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242883    2582 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242886    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242888    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242891    2582 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242894    2582 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242896    2582 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242899    2582 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242903    2582 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242905    2582 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242908    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:54:31.249668 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242910    2582 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242913    2582 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242915    2582 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242918    2582 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242920    2582 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242923    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242927    2582 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242929    2582 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242932    2582 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242934    2582 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242937    2582 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242939    2582 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242942    2582 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242944    2582 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242947    2582 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242949    2582 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242952    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242955    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242957    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242960    2582 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:54:31.250177 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242963    2582 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242966    2582 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242969    2582 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242972    2582 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242974    2582 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242977    2582 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242979    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242982    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242985    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242989    2582 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242991    2582 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242994    2582 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242996    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.242999    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243001    2582 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243004    2582 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243007    2582 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243010    2582 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243014    2582 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:54:31.250632 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243017    2582 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243020    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243022    2582 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243025    2582 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243027    2582 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243030    2582 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243032    2582 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243035    2582 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243037    2582 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243040    2582 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243042    2582 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243045    2582 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243047    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243050    2582 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243053    2582 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243055    2582 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243058    2582 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243061    2582 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243063    2582 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243067    2582 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:54:31.251121 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243069    2582 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243072    2582 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243075    2582 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243078    2582 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243080    2582 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243084    2582 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243086    2582 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243088    2582 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:31.243091    2582 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.243158    2582 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.246625    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.247936    2582 server.go:1014] "Starting client certificate rotation"
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.248039    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.248084    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 05:54:31.251602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.250381    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 05:54:31.252588 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.252568    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 05:54:31.263982 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.263961    2582 log.go:25] "Validated CRI v1 runtime API"
Nov 03 05:54:31.267976 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.267957    2582 log.go:25] "Validated CRI v1 image API"
Nov 03 05:54:31.268826 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.268782    2582 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 05:54:31.271917 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.271893    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a05087a3-1960-4a88-8686-4b09d87fc7f4:/dev/nvme0n1p3 c86b0c0d-73a6-4d06-8be4-41849362e17d:/dev/nvme0n1p4]
Nov 03 05:54:31.272006 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.271915    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 05:54:31.276528 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.276399    2582 manager.go:217] Machine: {Timestamp:2026-11-03 05:54:31.275925106 +0000 UTC m=+0.366369883 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3200463 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292e8f39fc025f09a4753cbc6b0655 SystemUUID:ec292e8f-39fc-025f-09a4-753cbc6b0655 BootID:5c55df60-70a7-45c5-8229-21f10a76eceb Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:d5:1d:c4:03 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:d5:1d:c4:03 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:d6:2e:d6:5e:6a:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 05:54:31.277109 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.277097    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 05:54:31.277199 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.277187    2582 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 05:54:31.280717 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280683    2582 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 05:54:31.280876 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280718    2582 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 05:54:31.280928 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280880    2582 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 05:54:31.280928 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280887    2582 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 05:54:31.281020 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280935    2582 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 05:54:31.281020 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.280955    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 05:54:31.282091 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.282079    2582 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 05:54:31.285533 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.285518    2582 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 05:54:31.285669 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.285659    2582 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 05:54:31.286581 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.286570    2582 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 05:54:31.286619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.286586    2582 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 05:54:31.286619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.286598    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 05:54:31.286619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.286607    2582 kubelet.go:405] "Adding apiserver pod source"
Nov 03 05:54:31.286619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.286617    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 05:54:31.287038 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.287011    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 05:54:31.289605 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.289581    2582 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 05:54:31.295972 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.295953    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 05:54:31.296953 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.296940    2582 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 05:54:31.297016 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.296970    2582 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 05:54:31.297166 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297146    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 05:54:31.297166 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297170    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 05:54:31.297281 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297177    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 05:54:31.297281 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297182    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 05:54:31.297281 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297190    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 05:54:31.297281 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297198    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 05:54:31.297945 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297935    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 05:54:31.297975 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297946    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 05:54:31.297975 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297953    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 05:54:31.297975 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297959    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 05:54:31.297975 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297973    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 05:54:31.298132 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.297982    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 05:54:31.299876 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.299866    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 05:54:31.299876 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.299876    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 05:54:31.303888 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.303875    2582 server.go:1263] "Started kubelet"
Nov 03 05:54:31.303968 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.303937    2582 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 05:54:31.304053 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.303996    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 05:54:31.304098 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.304083    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 05:54:31.304786 ip-10-0-132-233 systemd[1]: Started Kubernetes Kubelet.
Nov 03 05:54:31.305848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.305301    2582 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 05:54:31.306201 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.306186    2582 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 05:54:31.310924 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.310900    2582 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 05:54:31.314745 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.314725    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 05:54:31.315309 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.315290    2582 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 05:54:31.315500 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.315479    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.316076 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316057    2582 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 05:54:31.316076 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316078    2582 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 05:54:31.316230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316058    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 05:54:31.316230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316180    2582 factory.go:55] Registering systemd factory
Nov 03 05:54:31.316230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316194    2582 factory.go:223] Registration of the systemd container factory successfully
Nov 03 05:54:31.316230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316206    2582 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 05:54:31.316230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316212    2582 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 05:54:31.316457 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316430    2582 factory.go:153] Registering CRI-O factory
Nov 03 05:54:31.316502 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316459    2582 factory.go:223] Registration of the crio container factory successfully
Nov 03 05:54:31.316534 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316513    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 05:54:31.316534 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316526    2582 factory.go:103] Registering Raw factory
Nov 03 05:54:31.316589 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.316539    2582 manager.go:1199] Started watching for new ooms in manager
Nov 03 05:54:31.317572 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.317559    2582 manager.go:319] Starting recovery of all containers
Nov 03 05:54:31.330578 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.330462    2582 manager.go:324] Recovery completed
Nov 03 05:54:31.335004 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.334987    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.337056 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337030    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.337142 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337075    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.337142 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337089    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.337678 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337664    2582 cpu_manager.go:226] "Starting" policy="none"
Nov 03 05:54:31.337747 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337676    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.337832 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337675    2582 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 05:54:31.337832 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.337778    2582 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 05:54:31.338588 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.338574    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.338646 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.338600    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.338646 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.338613    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.339177 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.339164    2582 policy_none.go:50] "Start"
Nov 03 05:54:31.339246 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.339189    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 05:54:31.339246 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.339202    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 05:54:31.340256 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.340243    2582 policy_none.go:44] "Start"
Nov 03 05:54:31.343704 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.343679    2582 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-132-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 05:54:31.343817 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.343699    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-132-233.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 05:54:31.345031 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.343681    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-233.ec2.internal.18e473403aea7abd  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-233.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-132-233.ec2.internal,},FirstTimestamp:2026-11-03 05:54:31.303846589 +0000 UTC m=+0.394291367,LastTimestamp:2026-11-03 05:54:31.303846589 +0000 UTC m=+0.394291367,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-233.ec2.internal,}"
Nov 03 05:54:31.350985 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.350959    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 05:54:31.364622 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.364518    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-233.ec2.internal.18e473403ce53b60  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-233.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-132-233.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-132-233.ec2.internal,},FirstTimestamp:2026-11-03 05:54:31.33705712 +0000 UTC m=+0.427501897,LastTimestamp:2026-11-03 05:54:31.33705712 +0000 UTC m=+0.427501897,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-233.ec2.internal,}"
Nov 03 05:54:31.364742 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.364688    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 05:54:31.364742 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.364594    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-132-233.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 05:54:31.364742 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.364706    2582 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.375609    2582 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.379662    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q8f7l"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.382632    2582 manager.go:341] "Starting Device Plugin manager"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.382664    2582 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.382673    2582 server.go:87] "Starting device plugin registration server"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.382979    2582 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.382991    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.383097    2582 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.383174    2582 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.383181    2582 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.383781    2582 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 05:54:31.388841 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.383849    2582 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.389027    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.390024    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-q8f7l"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.390193    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.390211    2582 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.390228    2582 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 05:54:31.390725 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.390265    2582 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 05:54:31.391811 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.391705    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-132-233.ec2.internal.18e473403ce59acc  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-132-233.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-132-233.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-132-233.ec2.internal,},FirstTimestamp:2026-11-03 05:54:31.337081548 +0000 UTC m=+0.427526324,LastTimestamp:2026-11-03 05:54:31.337081548 +0000 UTC m=+0.427526324,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-132-233.ec2.internal,}"
Nov 03 05:54:31.424187 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.424019    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:54:31.483664 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.483587    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.484540 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.484519    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.484649 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.484559    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.484649 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.484573    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.484649 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.484602    2582 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.491294 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.491273    2582 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"]
Nov 03 05:54:31.491377 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.491331    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.492310 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.492293    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.492378 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.492325    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.492378 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.492341    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.497165 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.495753    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.497165 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.495959    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.497165 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.496022    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.498031 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498011    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.498115 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498015    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.498115 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498062    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.498115 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498072    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.498115 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498042    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.498231 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.498120    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.500050 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.500035    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.500095 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.500063    2582 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:54:31.500858 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.500842    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:54:31.500931 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.500871    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:54:31.500931 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.500884    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:54:31.505711 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.505695    2582 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.505781 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.505717    2582 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-132-233.ec2.internal\": node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.517481 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.517456    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-233.ec2.internal\" not found" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.520610 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.520593    2582 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-132-233.ec2.internal\" not found" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.555012 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.554982    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.617019 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.616979    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.617208 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.617062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.617208 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.617135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.655686 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.655648    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.717426 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717393    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.717573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.717573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.717573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717505    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.717573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717506    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.717573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.717553    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.756644 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.756578    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.822525 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.822488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.824611 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:31.824593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:31.856864 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.856823    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:31.957321 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:31.957283    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:32.060449 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:32.060419    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:32.162500 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:32.162457    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:32.247311 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.247277    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 05:54:32.247896 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.247441    2582 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 05:54:32.262925 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:32.262846    2582 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:54:32.330054 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:32.330019    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc364535d6718828aba0cd8493dd684.slice/crio-0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d WatchSource:0}: Error finding container 0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d: Status 404 returned error can't find the container with id 0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d
Nov 03 05:54:32.330332 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:32.330308    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b9c97e9d8c2351046139958cd95120b.slice/crio-5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73 WatchSource:0}: Error finding container 5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73: Status 404 returned error can't find the container with id 5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73
Nov 03 05:54:32.340710 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.340694    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 05:54:32.362008 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.361981    2582 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:54:32.373677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.373657    2582 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:54:32.393373 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.393327    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerStarted","Data":"0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d"}
Nov 03 05:54:32.394319 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.394295    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 05:49:31 +0000 UTC" deadline="2028-05-27 23:03:19.203181297 +0000 UTC"
Nov 03 05:54:32.394319 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.394319    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13721h8m46.8088668s"
Nov 03 05:54:32.394435 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.394331    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal" event={"ID":"1b9c97e9d8c2351046139958cd95120b","Type":"ContainerStarted","Data":"5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73"}
Nov 03 05:54:32.415669 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.415646    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:32.447925 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.447903    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:54:32.450318 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.450301    2582 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:54:32.473185 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.473166    2582 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:54:32.655594 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:32.655512    2582 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:54:33.287102 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.287070    2582 apiserver.go:52] "Watching apiserver"
Nov 03 05:54:33.316452 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.316413    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 05:54:33.322122 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.322082    2582 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 05:54:33.322511 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.322482    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bws8t","openshift-multus/network-metrics-daemon-glzh2","openshift-network-diagnostics/network-check-target-g5slj","openshift-network-operator/iptables-alerter-lgjdg","kube-system/global-pull-secret-syncer-bxj2x","openshift-cluster-node-tuning-operator/tuned-48zrq","openshift-dns/node-resolver-hvprb","kube-system/konnectivity-agent-xbkcj","openshift-multus/multus-2zxms","openshift-ovn-kubernetes/ovnkube-node-hthhm","kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal","openshift-multus/multus-additional-cni-plugins-lgrkn","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"]
Nov 03 05:54:33.324751 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.324726    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.326446 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.326424    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.328250 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.328228    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:33.328361 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.328298    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:33.330152 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.330124    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.331977 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.331877    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.331977 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.331940    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:33.335243 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.335221    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.337041 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.337021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.339379 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.339263    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.345085 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345061    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.345200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345084    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.345200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345151    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.345309 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345284    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.345360 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345319    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nphqj\""
Nov 03 05:54:33.345360 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.345341    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:33.345468 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.345396    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfnmx\""
Nov 03 05:54:33.347413 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.347317    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 05:54:33.347413 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.347369    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 05:54:33.347413 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.347385    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 05:54:33.347630 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.347583    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.350694 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.350655    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.350694 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.350661    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-75slc\""
Nov 03 05:54:33.350868 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.350724    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.350929 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.350891    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.351107 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.351087    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 05:54:33.351936 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.351567    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.355286 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.355258    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.377082 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.377044    2582 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 05:54:33.377227 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.377100    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsdjw"
Nov 03 05:54:33.389426 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.389400    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-vsdjw"
Nov 03 05:54:33.417512 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.417478    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 05:54:33.429065 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6wrz\" (UniqueName: \"kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429129    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snfcp\" (UniqueName: \"kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429172    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429197    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.429249 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429240    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429302    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429358    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429390    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429456    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429495    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb64g\" (UniqueName: \"kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.429596 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429611    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429639    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429663    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429688    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429725    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429775    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429863    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429918    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429949    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.429984    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.430003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430002    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430063    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430124    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2kgp\" (UniqueName: \"kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.430391 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430151    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.430689 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.430772 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430705    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430772 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430733    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430772 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430760    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430834    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.430943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430878    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.430943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430921    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.430943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430937    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430952    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.430981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431008    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431034    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431055    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431071    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431125    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.431189 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431242    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431268    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431301    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbbhl\" (UniqueName: \"kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431329    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431351    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431383    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431407    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sp7\" (UniqueName: \"kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431426    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431516    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431537    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431557    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.431704 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431631    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76kc7\" (UniqueName: \"kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431681    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431702    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431726    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431756    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431786    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431810    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr868\" (UniqueName: \"kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.432370 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.431847    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453611    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453685    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453706    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453771    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453781    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453781    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453773    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 05:54:33.453848 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453820    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z796t\""
Nov 03 05:54:33.454325 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453861    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.454325 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453870    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 05:54:33.454325 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.453952    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 05:54:33.454325 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454083    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 05:54:33.454325 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454290    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 05:54:33.454498 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454365    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krwzb\""
Nov 03 05:54:33.454498 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454382    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nvbtr\""
Nov 03 05:54:33.454498 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454430    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.454498 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.454469    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 05:54:33.470230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.469738    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 05:54:33.470230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.469777    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fkxng\""
Nov 03 05:54:33.470230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.469927    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8hrbq\""
Nov 03 05:54:33.470230 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.470077    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 05:54:33.471107 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.471078    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq86f\""
Nov 03 05:54:33.471607 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.471584    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 05:54:33.533020 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.532983    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533020 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533022    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533279 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfcp\" (UniqueName: \"kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533279 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533279 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533139    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533279 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533224    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533279 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533244    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533322    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533346    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533386    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533437 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533461    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533481    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533492    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb64g\" (UniqueName: \"kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533538    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533603    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533618    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533647 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533638    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533654    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533683    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533699    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533713    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533755    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533769    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533784    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533800    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kgp\" (UniqueName: \"kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533870    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533871    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533909    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533924    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.533997 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533932    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533940    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533952    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.533972    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533988    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534016    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.534037    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.034017975 +0000 UTC m=+3.124462756 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534041    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534055    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534097    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534097    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534214    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534242    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534301    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.533881    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534331    2582 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 05:54:33.534486 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534347    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534373    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534421    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534444    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534461    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534487    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534508    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534503    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534591    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534605    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534615    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534629    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534652    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534701    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.535121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534727    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534751    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534773    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534797    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534823    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534847    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534893    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534907    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534921    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534965    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534980    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534995    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535040    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535067    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbhl\" (UniqueName: \"kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535071    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.535079    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:33.535808 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535109    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535124    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.535138    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.03512176 +0000 UTC m=+3.125566558 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535143    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535148    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535162    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535165    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535243    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535245    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sp7\" (UniqueName: \"kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535258    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535277    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535298    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535364    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535428    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535454    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.536677 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535477    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535480    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.534337    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535552    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535619    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535668    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kc7\" (UniqueName: \"kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535760    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535810    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535908    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr868\" (UniqueName: \"kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.535934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.537575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wrz\" (UniqueName: \"kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536296    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536665    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536830    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536866    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.536912    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537046    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537089    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537131    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537172    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537210    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537246    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537388    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537426    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537471    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537519    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.538491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.537562    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.539196 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.538513    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.539196 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.538660    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.539196 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.538784    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.539196 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.538814    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.539196 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.539130    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.539357 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.539198    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.541270 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.541245    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.561686 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.561655    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb64g\" (UniqueName: \"kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.568172 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.568133    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfcp\" (UniqueName: \"kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.568299 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.568230    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.571484 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.571315    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:33.571484 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.571347    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:33.571484 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.571361    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:33.571484 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:33.571445    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:54:34.071424034 +0000 UTC m=+3.161868818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:33.573567 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.573541    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kgp\" (UniqueName: \"kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.574162 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.574098    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wrz\" (UniqueName: \"kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.574293 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.574271    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sp7\" (UniqueName: \"kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.578684 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.578603    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbhl\" (UniqueName: \"kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.579106 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.579056    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr868\" (UniqueName: \"kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.579290 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.579264    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kc7\" (UniqueName: \"kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:33.639012 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.638971    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:54:33.643470 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.643447    2582 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:54:33.649611 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.649581    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81d514bd_df1e_428c_ab18_66f841c88928.slice/crio-0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21 WatchSource:0}: Error finding container 0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21: Status 404 returned error can't find the container with id 0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21
Nov 03 05:54:33.661550 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.661520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:54:33.665718 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.665691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:54:33.667755 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.667736    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:54:33.677085 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.677062    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2zxms"
Nov 03 05:54:33.686587 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.686556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:54:33.709338 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.709297    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc82bd6b6_edf0_4def_8b73_969955c6e014.slice/crio-9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e WatchSource:0}: Error finding container 9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e: Status 404 returned error can't find the container with id 9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e
Nov 03 05:54:33.711085 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.710007    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23536828_f0dc_4f42_8f23_988d8de6a262.slice/crio-54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8 WatchSource:0}: Error finding container 54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8: Status 404 returned error can't find the container with id 54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8
Nov 03 05:54:33.712774 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.712711    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:33.713281 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.713212    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d92dbe9_16b1_4412_a14d_a2e7367d169a.slice/crio-a0e84ff3c9fd79724714fdc70f1a4e01884de373165582bac6faf42db60d7bd9 WatchSource:0}: Error finding container a0e84ff3c9fd79724714fdc70f1a4e01884de373165582bac6faf42db60d7bd9: Status 404 returned error can't find the container with id a0e84ff3c9fd79724714fdc70f1a4e01884de373165582bac6faf42db60d7bd9
Nov 03 05:54:33.713520 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.713494    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd8c881e_4142_4e06_9053_8cc113417d69.slice/crio-b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b WatchSource:0}: Error finding container b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b: Status 404 returned error can't find the container with id b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b
Nov 03 05:54:33.714880 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.714856    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37062547_bd74_437f_a325_dbb4a09bffe1.slice/crio-42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb WatchSource:0}: Error finding container 42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb: Status 404 returned error can't find the container with id 42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb
Nov 03 05:54:33.724432 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.724401    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:33.724942 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.724918    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac9a7a9_12dc_4699_bf59_da9a169f261d.slice/crio-24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9 WatchSource:0}: Error finding container 24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9: Status 404 returned error can't find the container with id 24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9
Nov 03 05:54:33.725742 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:33.725697    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:54:33.739677 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.739635    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2b850b_029d_42d2_aeb0_3856d68104f8.slice/crio-e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e WatchSource:0}: Error finding container e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e: Status 404 returned error can't find the container with id e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e
Nov 03 05:54:33.740772 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:54:33.740720    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podced67fea_5376_4c80_81cd_bdb12cccbd53.slice/crio-e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1 WatchSource:0}: Error finding container e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1: Status 404 returned error can't find the container with id e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1
Nov 03 05:54:34.040248 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.040218    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:34.040443 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.040271    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:34.040443 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.040417    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:34.040443 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.040420    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:34.040609 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.040480    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.040460509 +0000 UTC m=+4.130905296 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:34.040609 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.040494    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.040487908 +0000 UTC m=+4.130932672 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:34.142052 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.141117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:34.142052 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.141521    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:34.142052 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.141543    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:34.142052 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.141568    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:34.142052 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.141622    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:54:35.141605715 +0000 UTC m=+4.232050481 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:34.390982 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.390904    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:34.392014 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:34.391044    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:34.392606 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.392554    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:33 +0000 UTC" deadline="2028-06-26 06:37:50.00792325 +0000 UTC"
Nov 03 05:54:34.392606 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.392605    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14424h43m15.615321621s"
Nov 03 05:54:34.408520 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.408478    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbkcj" event={"ID":"cac9a7a9-12dc-4699-bf59-da9a169f261d","Type":"ContainerStarted","Data":"24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9"}
Nov 03 05:54:34.410739 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.410710    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerStarted","Data":"42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb"}
Nov 03 05:54:34.426242 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.425949    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lgjdg" event={"ID":"bd8c881e-4142-4e06-9053-8cc113417d69","Type":"ContainerStarted","Data":"b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b"}
Nov 03 05:54:34.434607 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.434557    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48zrq" event={"ID":"4d92dbe9-16b1-4412-a14d-a2e7367d169a","Type":"ContainerStarted","Data":"a0e84ff3c9fd79724714fdc70f1a4e01884de373165582bac6faf42db60d7bd9"}
Nov 03 05:54:34.451895 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.451858    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bws8t" event={"ID":"23536828-f0dc-4f42-8f23-988d8de6a262","Type":"ContainerStarted","Data":"54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8"}
Nov 03 05:54:34.472255 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.472221    2582 generic.go:356] "Generic (PLEG): container finished" podID="9bc364535d6718828aba0cd8493dd684" containerID="e4e8886420e10c5a4d141729a9f2e80f2eda333c21b68dc7768627ac06ad1984" exitCode=0
Nov 03 05:54:34.472493 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.472350    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerDied","Data":"e4e8886420e10c5a4d141729a9f2e80f2eda333c21b68dc7768627ac06ad1984"}
Nov 03 05:54:34.480488 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.480458    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2zxms" event={"ID":"c82bd6b6-edf0-4def-8b73-969955c6e014","Type":"ContainerStarted","Data":"9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e"}
Nov 03 05:54:34.495938 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.495907    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvprb" event={"ID":"81d514bd-df1e-428c-ab18-66f841c88928","Type":"ContainerStarted","Data":"0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21"}
Nov 03 05:54:34.510128 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.510096    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal" event={"ID":"1b9c97e9d8c2351046139958cd95120b","Type":"ContainerStarted","Data":"021f03637d7c470d861b14c404f7c15dae50047214fdee32179ffd615f4533a9"}
Nov 03 05:54:34.522734 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.522692    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4" event={"ID":"ced67fea-5376-4c80-81cd-bdb12cccbd53","Type":"ContainerStarted","Data":"e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1"}
Nov 03 05:54:34.540933 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:34.540784    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e"}
Nov 03 05:54:35.050379 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.050293    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:35.050379 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.050375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:35.050581 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.050505    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:35.050581 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.050565    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.050544783 +0000 UTC m=+6.140989568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:35.050687 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.050626    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:35.050687 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.050656    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.050645956 +0000 UTC m=+6.141090723 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:35.151760 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.151730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:35.151960 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.151881    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:35.151960 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.151900    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:35.151960 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.151911    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:35.152131 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.151973    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:54:37.151954425 +0000 UTC m=+6.242399214 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:35.393872 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.391903    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:35.393872 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.392043    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:35.393872 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.392484    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:35.393872 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:35.392579    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:35.395295 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.394533    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:33 +0000 UTC" deadline="2028-07-18 14:28:03.73795967 +0000 UTC"
Nov 03 05:54:35.395295 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.394561    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14960h33m28.343403281s"
Nov 03 05:54:35.553995 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.553900    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerStarted","Data":"b37a6db27cae845a576873eb6d4224ed11a29f322c45de6dcf6bcb45f9adc6ba"}
Nov 03 05:54:35.600887 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.600824    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal" podStartSLOduration=3.6007883830000003 podStartE2EDuration="3.600788383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:34.561151414 +0000 UTC m=+3.651596201" watchObservedRunningTime="2026-11-03 05:54:35.600788383 +0000 UTC m=+4.691233175"
Nov 03 05:54:35.601074 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:35.600921    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" podStartSLOduration=3.600916072 podStartE2EDuration="3.600916072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:35.600064694 +0000 UTC m=+4.690509481" watchObservedRunningTime="2026-11-03 05:54:35.600916072 +0000 UTC m=+4.691360859"
Nov 03 05:54:36.390557 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:36.390520    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:36.390746 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:36.390653    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:36.564862 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:36.563109    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="70efa738cd19217725558195123b5dfd6924c9a55bc7e458de9975dace0f65e3" exitCode=0
Nov 03 05:54:36.564862 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:36.564182    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"70efa738cd19217725558195123b5dfd6924c9a55bc7e458de9975dace0f65e3"}
Nov 03 05:54:37.069541 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.069372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:37.069541 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.069442    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:37.069783 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.069586    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:37.069783 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.069644    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.069627869 +0000 UTC m=+10.160072637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:37.070223 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.070053    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:37.070223 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.070101    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.070086388 +0000 UTC m=+10.160531165 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:37.169990 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.169949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:37.170161 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.170143    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:37.170215 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.170171    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:37.170215 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.170185    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:37.170297 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.170245    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:54:41.170227021 +0000 UTC m=+10.260671817 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:37.396005 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.395975    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:37.396145 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.396105    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:37.396311 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.396184    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:37.396311 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:37.396289    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:37.569753 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:37.569714    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xbkcj" event={"ID":"cac9a7a9-12dc-4699-bf59-da9a169f261d","Type":"ContainerStarted","Data":"1c194b9527392d905baf99a33223010d64e47889bcc4a74f8d1d02c71987ea7b"}
Nov 03 05:54:38.390963 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:38.390927    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:38.391141 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:38.391063    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:39.391579 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.391028    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:39.391579 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:39.391165    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:39.392179 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.391619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:39.392179 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:39.391738    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:39.576805 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.576697    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4" event={"ID":"ced67fea-5376-4c80-81cd-bdb12cccbd53","Type":"ContainerStarted","Data":"b0c4b0baf965483ac264692f1cbcfdc54182ddb2e13fc584be411241487161c5"}
Nov 03 05:54:39.579360 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.578644    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-bws8t" event={"ID":"23536828-f0dc-4f42-8f23-988d8de6a262","Type":"ContainerStarted","Data":"35f7f05bb649609428473dbb9b37a3b53eb5e89c187257d239414e9ad6ff2a5b"}
Nov 03 05:54:39.621555 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.621494    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xbkcj" podStartSLOduration=5.123940001 podStartE2EDuration="8.621461469s" totalImagesPullingTime="3.497521468s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:37.613738815 +0000 UTC m=+6.704183605" watchObservedRunningTime="2026-11-03 05:54:39.621461469 +0000 UTC m=+8.711906256"
Nov 03 05:54:39.986834 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.986771    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:39.988087 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:39.987836    2582 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:40.031683 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:40.030807    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-bws8t" podStartSLOduration=4.068274458 podStartE2EDuration="9.030769131s" totalImagesPullingTime="4.962494673s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:39.6224747 +0000 UTC m=+8.712919486" watchObservedRunningTime="2026-11-03 05:54:40.030769131 +0000 UTC m=+9.121213917"
Nov 03 05:54:40.390947 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:40.390849    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:40.391110 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:40.390990    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:40.581110 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:40.581081    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:40.581815 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:40.581770    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:54:40.694029 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:40.693996    2582 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 05:54:41.104497 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.104459    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:41.104629 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.104529    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:41.104703 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.104662    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:41.104762 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.104712    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.104697226 +0000 UTC m=+18.195142003 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:41.105055 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.105034    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:41.105146 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.105080    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.10506962 +0000 UTC m=+18.195514396 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:41.206390 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.205874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:41.206390 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.206029    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:41.206390 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.206047    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:41.206390 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.206058    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:41.206390 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.206106    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:54:49.206093748 +0000 UTC m=+18.296538524 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:41.388211 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.388087    2582 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T05:54:40.694021872Z","UUID":"8f1a925d-e91c-4050-9635-23b342c3e986","Handler":null,"Name":"","Endpoint":""}
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.390831    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.390909    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.391636    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.391740    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.392593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:41.392746 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:41.392701    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:41.585627 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.584970    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2zxms" event={"ID":"c82bd6b6-edf0-4def-8b73-969955c6e014","Type":"ContainerStarted","Data":"1def756b02122e5028701c77cb4ffdc646a5a9e5f4b60c06649916901e1369b3"}
Nov 03 05:54:41.588022 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.587986    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4" event={"ID":"ced67fea-5376-4c80-81cd-bdb12cccbd53","Type":"ContainerStarted","Data":"0ead8338b39fbf128aa2d3d82862fd577e0d6278c3a68a1a9bb7fc0a68ec4ed1"}
Nov 03 05:54:41.598408 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.598378    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="30c2b02d20f93680fe7cbc7ea15c2f02ef164999f704544bce49edcee1c02f65" exitCode=0
Nov 03 05:54:41.598901 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.598873    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"30c2b02d20f93680fe7cbc7ea15c2f02ef164999f704544bce49edcee1c02f65"}
Nov 03 05:54:41.635007 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:41.634741    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2zxms" podStartSLOduration=3.081667368 podStartE2EDuration="10.634724043s" totalImagesPullingTime="7.553056675s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:41.634301551 +0000 UTC m=+10.724746334" watchObservedRunningTime="2026-11-03 05:54:41.634724043 +0000 UTC m=+10.725168828"
Nov 03 05:54:42.390898 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.390703    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:42.390898 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:42.390830    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:42.602436 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.602401    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hvprb" event={"ID":"81d514bd-df1e-428c-ab18-66f841c88928","Type":"ContainerStarted","Data":"523f242dc78736a805376e7df94519aa7ee1a04fa9b6b4ba5bcab8ad5b3ae33e"}
Nov 03 05:54:42.605030 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.604940    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4" event={"ID":"ced67fea-5376-4c80-81cd-bdb12cccbd53","Type":"ContainerStarted","Data":"6ec05bd4126cd18c3d1181963f9d62e27e2bfbbefcb38628e1c333e43b1860b2"}
Nov 03 05:54:42.607083 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.607059    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="219c096ec1354322c072ebfbd276688a7af6d8778dd4d3fb541fac7b0772d575" exitCode=0
Nov 03 05:54:42.607208 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.607144    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"219c096ec1354322c072ebfbd276688a7af6d8778dd4d3fb541fac7b0772d575"}
Nov 03 05:54:42.646308 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.646214    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hvprb" podStartSLOduration=3.867393694 podStartE2EDuration="11.646198238s" totalImagesPullingTime="7.778804544s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:42.646080904 +0000 UTC m=+11.736525689" watchObservedRunningTime="2026-11-03 05:54:42.646198238 +0000 UTC m=+11.736643023"
Nov 03 05:54:42.692000 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:42.691949    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4" podStartSLOduration=3.091876343 podStartE2EDuration="10.691935035s" totalImagesPullingTime="7.600058692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:32 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:42.691874952 +0000 UTC m=+11.782319737" watchObservedRunningTime="2026-11-03 05:54:42.691935035 +0000 UTC m=+11.782379820"
Nov 03 05:54:43.390656 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.390617    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:43.390810 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.390661    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:43.390810 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:43.390755    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:43.390894 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:43.390876    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:43.611709 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.611676    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="26ecdcbca36c65c74c6eebb49c2bd8c0ed58eb4fbf6432600ab705b314006c89" exitCode=0
Nov 03 05:54:43.612221 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.611769    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"26ecdcbca36c65c74c6eebb49c2bd8c0ed58eb4fbf6432600ab705b314006c89"}
Nov 03 05:54:43.613752 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.613722    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-lgjdg" event={"ID":"bd8c881e-4142-4e06-9053-8cc113417d69","Type":"ContainerStarted","Data":"ede8412cd21d31bea9cbb4214e69d6f3c6088aaaf5b34dddfde6bdc901ec0d61"}
Nov 03 05:54:43.705494 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:43.705435    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-lgjdg" podStartSLOduration=5.063662877 podStartE2EDuration="12.705418491s" totalImagesPullingTime="7.641755614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:43.704736863 +0000 UTC m=+12.795181650" watchObservedRunningTime="2026-11-03 05:54:43.705418491 +0000 UTC m=+12.795863281"
Nov 03 05:54:44.390676 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:44.390634    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:44.390887 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:44.390775    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:44.618895 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:44.618855    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-48zrq" event={"ID":"4d92dbe9-16b1-4412-a14d-a2e7367d169a","Type":"ContainerStarted","Data":"9c2dc8adec9485251a8cb59a86be8899b90fccbcb4c94d057744e719fb31752a"}
Nov 03 05:54:44.663850 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:44.663729    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-48zrq" podStartSLOduration=3.7779796450000003 podStartE2EDuration="13.663715943s" totalImagesPullingTime="9.885736298s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:44.663567761 +0000 UTC m=+13.754012557" watchObservedRunningTime="2026-11-03 05:54:44.663715943 +0000 UTC m=+13.754160729"
Nov 03 05:54:45.391112 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:45.391021    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:45.391112 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:45.391066    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:45.391267 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:45.391147    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:45.391556 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:45.391532    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:45.622920 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:45.622833    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="efa623b1aee0c2f4497d16e96419228b9cb7923a8a40f4a3763fbea8990c734b" exitCode=0
Nov 03 05:54:45.623536 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:45.622929    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"efa623b1aee0c2f4497d16e96419228b9cb7923a8a40f4a3763fbea8990c734b"}
Nov 03 05:54:46.391573 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:46.391364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:46.391745 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:46.391665    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:46.629111 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:46.629059    2582 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="647a099ed233fdc4bfce94edfb3706c855d067b9ac452a38721134a948a09e8b" exitCode=0
Nov 03 05:54:46.629575 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:46.629148    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerDied","Data":"647a099ed233fdc4bfce94edfb3706c855d067b9ac452a38721134a948a09e8b"}
Nov 03 05:54:47.390726 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:47.390692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:47.390893 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:47.390853    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:47.390944 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:47.390907    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:47.391062 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:47.391020    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:47.635201 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:47.635166    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" event={"ID":"37062547-bd74-437f-a325-dbb4a09bffe1","Type":"ContainerStarted","Data":"e1efb09b08d22cbafc616af0ae70f993e95ac7a34ccfccfe023b4c09fc996092"}
Nov 03 05:54:47.714111 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:47.714014    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-lgrkn" podStartSLOduration=7.397456556 podStartE2EDuration="16.714000061s" totalImagesPullingTime="9.316543505s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:47.699500441 +0000 UTC m=+16.789945225" watchObservedRunningTime="2026-11-03 05:54:47.714000061 +0000 UTC m=+16.804444846"
Nov 03 05:54:48.390732 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.390710    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:48.390850 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:48.390817    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:48.640475 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.640290    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"c7a98c00ef375174ca0f4a064778dab970d15c939262d7c7604b520cde510659"}
Nov 03 05:54:48.640741 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.640523    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"76bbf8f200e09e656e9ba40995a115b302072a27662beb089124de5c5d9e8045"}
Nov 03 05:54:48.640741 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.640559    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"918fcf450883b25744af79affd23acf699a458317b3f0ef82efaa02a78564058"}
Nov 03 05:54:48.640741 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.640588    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"44c6ebb945f5dbec07825fe10c6dc966a38d83896398084345304c67f1d0c001"}
Nov 03 05:54:48.640741 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:48.640621    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"3ba7214bc8db9927746ad995e55cb6b558b695fd41631887b386cd73e6a699c9"}
Nov 03 05:54:49.168008 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.167979    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:49.168171 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.168033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:49.168171 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.168131    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:49.168267 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.168183    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.168170755 +0000 UTC m=+34.258615518 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 05:54:49.168267 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.168131    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:49.168331 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.168272    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret podName:6e5c002c-8b36-498b-aa36-c93aa0b3cb89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.168260625 +0000 UTC m=+34.258705402 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret") pod "global-pull-secret-syncer-bxj2x" (UID: "6e5c002c-8b36-498b-aa36-c93aa0b3cb89") : object "kube-system"/"original-pull-secret" not registered
Nov 03 05:54:49.269094 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.269061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:49.269246 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.269223    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 05:54:49.269282 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.269253    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 05:54:49.269282 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.269264    2582 projected.go:196] Error preparing data for projected volume kube-api-access-nrjg9 for pod openshift-network-diagnostics/network-check-target-g5slj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:49.269340 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.269313    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9 podName:cc744642-5f3e-43f7-a565-4db72473b5aa nodeName:}" failed. No retries permitted until 2026-11-03 05:55:05.26929886 +0000 UTC m=+34.359743637 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjg9" (UniqueName: "kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9") pod "network-check-target-g5slj" (UID: "cc744642-5f3e-43f7-a565-4db72473b5aa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 05:54:49.390935 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.390906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:49.391090 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.390913    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:49.391090 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.391023    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:49.391090 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:49.391071    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:49.644706 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:49.644634    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"57c0d2d40496f915caa45a2293e38c0ff49c06b7e684323e53b8745f64c70852"}
Nov 03 05:54:50.390831 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:50.390780    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:50.391002 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:50.390893    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:51.391990 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:51.391956    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:51.392559 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:51.392225    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:51.394430 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:51.392721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:51.394430 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:51.392875    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:51.652096 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:51.652024    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"9d884fa4a4cf8efb012edfbe11d8a759a4b8709bd7cae4118c6d559763fcbe14"}
Nov 03 05:54:52.391283 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:52.391240    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:52.391439 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:52.391343    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:53.390414 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.390388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:53.391060 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.390388    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:53.391060 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:53.390540    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:53.391060 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:53.390560    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:53.659692 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.659610    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" event={"ID":"7f2b850b-029d-42d2-aeb0-3856d68104f8","Type":"ContainerStarted","Data":"16749ceaffff981d83cffb3dbc5d69569122b4e7722667f004962804c4156a49"}
Nov 03 05:54:53.659878 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.659860    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:53.659942 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.659926    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:53.674737 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.674714    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:53.760951 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:53.760902    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm" podStartSLOduration=8.118227484 podStartE2EDuration="22.76088952s" totalImagesPullingTime="14.642662036s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:54:53.760587919 +0000 UTC m=+22.851032706" watchObservedRunningTime="2026-11-03 05:54:53.76088952 +0000 UTC m=+22.851334306"
Nov 03 05:54:54.391491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.391455    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:54.391886 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:54.391559    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:54.662247 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.662174    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:54.676944 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.676908    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:54:54.861416 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.861382    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5slj"]
Nov 03 05:54:54.861580 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.861498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:54.861624 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:54.861590    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:54.862295 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.862273    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bxj2x"]
Nov 03 05:54:54.862390 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.862367    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:54.862448 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:54.862434    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:54.862747 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.862730    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-glzh2"]
Nov 03 05:54:54.862840 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:54.862829    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:54.862915 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:54.862901    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:56.390741 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:56.390507    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:56.391221 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:56.390509    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:56.391221 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:56.390526    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:56.391221 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:56.390856    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:56.391221 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:56.390824    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:56.391221 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:56.390921    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:54:58.390966 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:58.390936    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:54:58.391453 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:58.390943    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:54:58.391453 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:54:58.391075    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:54:58.391453 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:58.391137    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:54:58.391453 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:58.391041    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:54:58.391453 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:54:58.391214    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:55:00.391369 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:00.391338    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:00.391771 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:00.391373    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:00.391771 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:00.391378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:00.391771 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:00.391447    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-bxj2x" podUID="6e5c002c-8b36-498b-aa36-c93aa0b3cb89"
Nov 03 05:55:00.391771 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:00.391541    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-glzh2" podUID="87a29b40-9e0d-4dde-bedb-c9423919e268"
Nov 03 05:55:00.391771 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:00.391616    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-g5slj" podUID="cc744642-5f3e-43f7-a565-4db72473b5aa"
Nov 03 05:55:02.248083 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.248057    2582 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeReady"
Nov 03 05:55:02.248502 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.248163    2582 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 05:55:02.310678 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.310650    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hxvcn"]
Nov 03 05:55:02.373205 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.373178    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p8jsb"]
Nov 03 05:55:02.373360 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.373295    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.385983 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.385962    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxvcn"]
Nov 03 05:55:02.385983 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.385993    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8jsb"]
Nov 03 05:55:02.386161 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.386113    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.390133 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390113    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 05:55:02.390299 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390278    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xj6m8\""
Nov 03 05:55:02.390299 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390282    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 05:55:02.390477 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390320    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 05:55:02.390477 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390362    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:02.390477 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390467    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:02.390601 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.390363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:02.450973 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.450938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.451130 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.450981    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.451130 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.451010    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.451130 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.451120    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vds6d\" (UniqueName: \"kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.451264 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.451152    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.451264 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.451193    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d2p4\" (UniqueName: \"kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.491389 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491356    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 05:55:02.491389 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491361    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 05:55:02.491619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491460    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 05:55:02.491619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491500    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rwdxr\""
Nov 03 05:55:02.491619 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491511    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 05:55:02.491728 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491625    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mlps4\""
Nov 03 05:55:02.491728 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491630    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 05:55:02.491728 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491631    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xvf66\""
Nov 03 05:55:02.491728 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491650    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 05:55:02.491728 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.491669    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwk29\""
Nov 03 05:55:02.552118 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vds6d\" (UniqueName: \"kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.552118 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.552118 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2p4\" (UniqueName: \"kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.552423 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.552423 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552186    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.552423 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552219    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.552423 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:02.552344    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:02.552423 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:02.552416    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:03.052397693 +0000 UTC m=+32.142842469 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:02.552653 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552537    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.552783 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.552765    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.555556 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.555540    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.581978 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.581942    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2p4\" (UniqueName: \"kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:02.583121 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.583096    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vds6d\" (UniqueName: \"kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:02.694691 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:02.694659    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:03.030426 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:03.030343    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hxvcn"]
Nov 03 05:55:03.033755 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:03.033729    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87999404_7a09_46b3_8d41_65407bb93863.slice/crio-3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f WatchSource:0}: Error finding container 3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f: Status 404 returned error can't find the container with id 3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f
Nov 03 05:55:03.056412 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:03.056377    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:03.056571 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:03.056551    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:03.056646 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:03.056626    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:04.056606202 +0000 UTC m=+33.147050968 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:03.682637 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:03.682598    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxvcn" event={"ID":"87999404-7a09-46b3-8d41-65407bb93863","Type":"ContainerStarted","Data":"3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f"}
Nov 03 05:55:04.063602 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:04.063519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:04.063757 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:04.063649    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:04.063757 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:04.063708    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:06.063692008 +0000 UTC m=+35.154136786 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:05.171893 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.171805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:05.172387 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.171898    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:05.172387 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:05.172038    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:05.172387 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:05.172104    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:37.172085144 +0000 UTC m=+66.262529920 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:05.175441 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.175401    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:05.273042 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.273004    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:05.275481 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.275447    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:05.442893 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.442814    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:05.458003 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.457976    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:05.903772 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.903737    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-g5slj"]
Nov 03 05:55:05.905383 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:05.905356    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc744642_5f3e_43f7_a565_4db72473b5aa.slice/crio-ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980 WatchSource:0}: Error finding container ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980: Status 404 returned error can't find the container with id ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980
Nov 03 05:55:05.924424 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:05.924402    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-bxj2x"]
Nov 03 05:55:05.925713 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:05.925688    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5c002c_8b36_498b_aa36_c93aa0b3cb89.slice/crio-799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184 WatchSource:0}: Error finding container 799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184: Status 404 returned error can't find the container with id 799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184
Nov 03 05:55:06.077966 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:06.077886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:06.078104 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:06.078008    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:06.078104 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:06.078060    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:10.078047765 +0000 UTC m=+39.168492528 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:06.694392 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:06.694347    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hxvcn" event={"ID":"87999404-7a09-46b3-8d41-65407bb93863","Type":"ContainerStarted","Data":"79aa40c2a020dfe02031fc603f1640cfeb9e145aea8e3fcef74d14bc1e652226"}
Nov 03 05:55:06.695857 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:06.695828    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5slj" event={"ID":"cc744642-5f3e-43f7-a565-4db72473b5aa","Type":"ContainerStarted","Data":"ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980"}
Nov 03 05:55:06.697194 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:06.697156    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-bxj2x" event={"ID":"6e5c002c-8b36-498b-aa36-c93aa0b3cb89","Type":"ContainerStarted","Data":"799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184"}
Nov 03 05:55:06.738290 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:06.738018    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hxvcn" podStartSLOduration=2.152320036 podStartE2EDuration="4.738005333s" totalImagesPullingTime="2.585685297s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:06.737535544 +0000 UTC m=+35.827980331" watchObservedRunningTime="2026-11-03 05:55:06.738005333 +0000 UTC m=+35.828450116"
Nov 03 05:55:08.276275 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.276217    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5v8g7"]
Nov 03 05:55:08.313961 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.313924    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v8g7"]
Nov 03 05:55:08.314112 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.314045    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.330393 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.330364    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 05:55:08.330534 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.330475    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9vz5\""
Nov 03 05:55:08.330534 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.330521    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 05:55:08.345138 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.345112    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 05:55:08.345259 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.345185    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 05:55:08.356977 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.356952    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:55:08.374806 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.374768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.378300 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.378273    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:55:08.397770 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.397740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.397916 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.397833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dkk8\" (UniqueName: \"kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.397916 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.397868    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.398024 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.397913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.398024 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.397960    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.428821 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.428688    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 05:55:08.428821 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.428776    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7glbz\""
Nov 03 05:55:08.445261 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.445214    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 05:55:08.498807 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498770    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.498988 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498845    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.498988 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498878    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.498988 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.498988 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.498988 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.498967    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.499243 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.499012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.499243 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.499052    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkk8\" (UniqueName: \"kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.499243 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.499074    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.499243 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.499171    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.499243 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:08.499233    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:08.499443 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:08.499287    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:08.999270405 +0000 UTC m=+38.089715172 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:08.499443 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.499385    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.499582 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:08.499553    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 05:55:08.499637 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:08.499606    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:08.999589592 +0000 UTC m=+38.090034372 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : configmap "kube-rbac-proxy" not found
Nov 03 05:55:08.524812 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.524764    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkk8\" (UniqueName: \"kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:08.600468 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.600392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.600468 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.600444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.601200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.600490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.601200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.600548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.601200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.601065    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.601200 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.601139    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.601495 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.601475    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.631089 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.631064    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.698650 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.698593    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"]
Nov 03 05:55:08.708338 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.708310    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:08.730649 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.730619    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"]
Nov 03 05:55:08.730844 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.730748    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:08.756114 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.756093    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jcdvc\""
Nov 03 05:55:08.802265 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.802237    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgx64\" (UniqueName: \"kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64\") pod \"migrator-66df5d496b-hhsb4\" (UID: \"19e4e8ea-dd59-4bcb-924c-870fd4392bcf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:08.846877 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.846848    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 05:55:08.846877 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.846855    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 05:55:08.903327 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.903248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgx64\" (UniqueName: \"kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64\") pod \"migrator-66df5d496b-hhsb4\" (UID: \"19e4e8ea-dd59-4bcb-924c-870fd4392bcf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:08.932868 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:08.932840    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgx64\" (UniqueName: \"kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64\") pod \"migrator-66df5d496b-hhsb4\" (UID: \"19e4e8ea-dd59-4bcb-924c-870fd4392bcf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:09.004117 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.004084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:09.004309 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.004159    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:09.004309 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:09.004221    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 05:55:09.004430 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:09.004319    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:10.004301256 +0000 UTC m=+39.094746043 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : configmap "kube-rbac-proxy" not found
Nov 03 05:55:09.004430 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:09.004365    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:09.004510 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:09.004432    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:10.004413269 +0000 UTC m=+39.094858047 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:09.039275 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.039250    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:55:09.040759 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:09.040735    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fdd1c9_6616_4e37_9b39_ec6dc3d0435b.slice/crio-b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919 WatchSource:0}: Error finding container b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919: Status 404 returned error can't find the container with id b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919
Nov 03 05:55:09.065788 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.065753    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:09.176899 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.176832    2582 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8gmvq"]
Nov 03 05:55:09.196259 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.196218    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.204462 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.204392    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8gmvq"]
Nov 03 05:55:09.213242 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.213213    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 05:55:09.213382 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.213356    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 05:55:09.213491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.213450    2582 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 05:55:09.213491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.213455    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 05:55:09.213491 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.213460    2582 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tn6jl\""
Nov 03 05:55:09.306934 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.306910    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.307268 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.306939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k5jk\" (UniqueName: \"kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.307268 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.307102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.376257 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.376215    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"]
Nov 03 05:55:09.378299 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:09.378269    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e4e8ea_dd59_4bcb_924c_870fd4392bcf.slice/crio-632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21 WatchSource:0}: Error finding container 632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21: Status 404 returned error can't find the container with id 632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21
Nov 03 05:55:09.408168 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.408128    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.408168 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.408171    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.408368 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.408194    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5jk\" (UniqueName: \"kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.408943 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.408914    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.410712 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.410693    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.433532 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.433479    2582 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5jk\" (UniqueName: \"kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.525592 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.525556    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:09.707817 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.707715    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-g5slj" event={"ID":"cc744642-5f3e-43f7-a565-4db72473b5aa","Type":"ContainerStarted","Data":"71fad5dc0edb24251a1861163660f093ed9416f1595f9903bbddf2724293c07e"}
Nov 03 05:55:09.708504 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.708480    2582 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:09.709935 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.709909    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4" event={"ID":"19e4e8ea-dd59-4bcb-924c-870fd4392bcf","Type":"ContainerStarted","Data":"632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21"}
Nov 03 05:55:09.711216 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.711175    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj" event={"ID":"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b","Type":"ContainerStarted","Data":"b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919"}
Nov 03 05:55:09.755772 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.755724    2582 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-g5slj" podStartSLOduration=35.376272881 podStartE2EDuration="38.755707221s" totalImagesPullingTime="3.37943434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:54:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:09.754436962 +0000 UTC m=+38.844881746" watchObservedRunningTime="2026-11-03 05:55:09.755707221 +0000 UTC m=+38.846152210"
Nov 03 05:55:09.842290 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:09.842258    2582 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-8gmvq"]
Nov 03 05:55:09.844150 ip-10-0-132-233 kubenswrapper[2582]: W1103 05:55:09.844122    2582 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a33ca8_05a6_48cb_9c4f_b990d1c7ca62.slice/crio-169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92 WatchSource:0}: Error finding container 169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92: Status 404 returned error can't find the container with id 169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92
Nov 03 05:55:10.020762 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:10.020673    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:10.020946 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:10.020771    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:10.020946 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.020831    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 03 05:55:10.020946 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.020904    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:10.020946 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.020908    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:12.020885934 +0000 UTC m=+41.111330720 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : configmap "kube-rbac-proxy" not found
Nov 03 05:55:10.021159 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.020961    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:12.02094909 +0000 UTC m=+41.111393854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:10.121875 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:10.121837    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:10.122039 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.122015    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:10.122154 ip-10-0-132-233 kubenswrapper[2582]: E1103 05:55:10.122139    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:18.122118825 +0000 UTC m=+47.212563592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:10.716938 ip-10-0-132-233 kubenswrapper[2582]: I1103 05:55:10.716866    2582 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq" event={"ID":"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62","Type":"ContainerStarted","Data":"169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92"}
Nov 03 05:55:11.522306 ip-10-0-132-233 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 05:55:11.535238 ip-10-0-132-233 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 05:55:11.535448 ip-10-0-132-233 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 05:55:11.535510 ip-10-0-132-233 systemd[1]: kubelet.service: Consumed 2.395s CPU time.
Nov 03 05:55:11.543455 ip-10-0-132-233 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 05:55:11.642376 ip-10-0-132-233 kubenswrapper[4528]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:55:11.642376 ip-10-0-132-233 kubenswrapper[4528]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 05:55:11.642376 ip-10-0-132-233 kubenswrapper[4528]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:55:11.642376 ip-10-0-132-233 kubenswrapper[4528]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 05:55:11.644440 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644356    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.644440 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644440    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644451    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644454    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644457    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644461    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644464    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644466    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644469    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644472    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644474    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644477    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644480    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644482    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644485    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644488    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644490    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644493    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644495    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644498    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644500    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.644512 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644508    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644510    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644513    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644516    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644519    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644522    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644525    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644527    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644530    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644533    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644535    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644538    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644540    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644543    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644545    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644548    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644551    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644553    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644557    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.645005 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644560    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644562    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644565    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644567    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644570    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644573    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644576    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644578    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644581    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644584    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644587    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644589    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644592    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644595    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644598    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644600    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644603    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644606    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644608    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.645472 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644611    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644614    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644616    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644619    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644623    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644628    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644630    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644634    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644637    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644639    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644642    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644644    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644647    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644650    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644652    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644655    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644658    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644661    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644664    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644666    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.645940 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644669    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644672    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644674    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644677    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644680    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644683    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644685    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644688    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644692    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644695    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644699    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644702    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644705    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644707    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644710    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644713    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644715    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644718    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644720    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644723    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.646436 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644727    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644730    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644732    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644735    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644738    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644740    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.644748    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645522    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645530    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645532    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645535    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645538    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645541    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645543    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645546    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645549    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645559    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645563    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645566    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.646930 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645569    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645572    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645575    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645578    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645581    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645584    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645587    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645591    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645595    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645598    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645601    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645603    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645606    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645609    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645612    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645614    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645617    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645619    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645622    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645625    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.647379 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645632    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645635    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645638    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645641    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645643    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645646    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645649    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645651    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645654    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645656    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645659    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645661    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645664    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645666    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645669    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645687    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645691    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645694    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645697    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645701    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.647849 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645704    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645707    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645709    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645712    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645715    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645718    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645720    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645723    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645726    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645728    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645731    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645733    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645736    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645744    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645747    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645749    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645752    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645754    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645757    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645759    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.648355 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645762    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645764    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645767    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645770    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645773    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645775    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645778    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645780    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645783    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645785    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645788    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645809    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645811    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645814    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645817    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645820    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645822    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645825    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645828    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.648844 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645831    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645833    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645836    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645838    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645841    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645844    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645847    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645853    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645856    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645858    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645861    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645863    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645866    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645868    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.645871    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646363    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646369    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646373    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646375    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.649288 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646378    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646382    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646385    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646389    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646391    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646394    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646397    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646400    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646402    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646405    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646407    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646410    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646416    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646418    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646421    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646423    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646426    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646429    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646431    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.649740 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646434    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646438    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646442    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646445    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646448    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646451    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646454    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646456    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646458    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646461    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646463    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646466    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646469    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646471    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646475    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646477    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646480    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646482    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646485    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.650207 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646487    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646490    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646492    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646495    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646497    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646500    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646503    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646506    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646509    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646511    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646513    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646516    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646518    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646521    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646524    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646527    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646529    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646532    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646534    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.650769 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646536    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646540    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646542    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646544    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646547    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646549    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646552    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646554    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646557    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646560    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646562    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646565    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646567    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646570    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646572    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646575    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646577    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646580    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646582    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646586    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.651540 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646588    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646591    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646594    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646596    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646599    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646601    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646604    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646606    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646608    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646611    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646614    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646617    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646619    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646622    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646624    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646628    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646632    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646635    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646638    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.652125 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646641    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646644    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646646    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646649    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646651    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.646654    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646737    4528 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646744    4528 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646751    4528 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646755    4528 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646759    4528 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646763    4528 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646769    4528 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646773    4528 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646776    4528 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646779    4528 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646782    4528 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646785    4528 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646803    4528 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646807    4528 flags.go:64] FLAG: --cgroup-root=""
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646810    4528 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646814    4528 flags.go:64] FLAG: --client-ca-file=""
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646817    4528 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 05:55:11.652598 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646820    4528 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646824    4528 flags.go:64] FLAG: --cluster-domain=""
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646827    4528 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646831    4528 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646834    4528 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646837    4528 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646841    4528 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646844    4528 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646847    4528 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646850    4528 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646853    4528 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646856    4528 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646859    4528 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646862    4528 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646865    4528 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646870    4528 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646873    4528 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646875    4528 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646878    4528 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646883    4528 flags.go:64] FLAG: --enable-server="true"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646886    4528 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646893    4528 flags.go:64] FLAG: --event-burst="100"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646896    4528 flags.go:64] FLAG: --event-qps="50"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646899    4528 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 05:55:11.653148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646902    4528 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646905    4528 flags.go:64] FLAG: --eviction-hard=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646910    4528 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646913    4528 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646916    4528 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646919    4528 flags.go:64] FLAG: --eviction-soft=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646922    4528 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646925    4528 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646928    4528 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646931    4528 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646934    4528 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646938    4528 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646941    4528 flags.go:64] FLAG: --feature-gates=""
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646945    4528 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646948    4528 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646951    4528 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646954    4528 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646957    4528 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646960    4528 flags.go:64] FLAG: --help="false"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646963    4528 flags.go:64] FLAG: --hostname-override="ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646966    4528 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646969    4528 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646972    4528 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646975    4528 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 05:55:11.653716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646979    4528 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646981    4528 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646984    4528 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646988    4528 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646991    4528 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646995    4528 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.646999    4528 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647001    4528 flags.go:64] FLAG: --kube-reserved=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647005    4528 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647007    4528 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647010    4528 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647013    4528 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647016    4528 flags.go:64] FLAG: --lock-file=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647019    4528 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647022    4528 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647025    4528 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647030    4528 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647033    4528 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647037    4528 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647040    4528 flags.go:64] FLAG: --logging-format="text"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647043    4528 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647046    4528 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647049    4528 flags.go:64] FLAG: --manifest-url=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647052    4528 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647059    4528 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 05:55:11.654394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647062    4528 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647066    4528 flags.go:64] FLAG: --max-pods="110"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647069    4528 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647072    4528 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647075    4528 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647078    4528 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647081    4528 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647084    4528 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647087    4528 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647093    4528 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647098    4528 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647101    4528 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647105    4528 flags.go:64] FLAG: --pod-cidr=""
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647108    4528 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647111    4528 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647114    4528 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647117    4528 flags.go:64] FLAG: --port="10250"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647120    4528 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647123    4528 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cf09f0733869f80a"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647126    4528 flags.go:64] FLAG: --qos-reserved=""
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647129    4528 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647132    4528 flags.go:64] FLAG: --register-node="true"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647135    4528 flags.go:64] FLAG: --register-with-taints=""
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647139    4528 flags.go:64] FLAG: --registry-burst="10"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647142    4528 flags.go:64] FLAG: --registry-qps="5"
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647145    4528 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 05:55:11.655070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647149    4528 flags.go:64] FLAG: --reserved-memory=""
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647152    4528 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647156    4528 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647158    4528 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647161    4528 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647164    4528 flags.go:64] FLAG: --runonce="false"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647167    4528 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647170    4528 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647173    4528 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647176    4528 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647179    4528 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647182    4528 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647185    4528 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647188    4528 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647191    4528 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647194    4528 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647198    4528 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647200    4528 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647205    4528 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647208    4528 flags.go:64] FLAG: --system-cgroups=""
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647210    4528 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647215    4528 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647218    4528 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647221    4528 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647225    4528 flags.go:64] FLAG: --tls-min-version=""
Nov 03 05:55:11.655700 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647228    4528 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647231    4528 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647233    4528 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647236    4528 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647239    4528 flags.go:64] FLAG: --v="2"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647244    4528 flags.go:64] FLAG: --version="false"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647248    4528 flags.go:64] FLAG: --vmodule=""
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647251    4528 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647255    4528 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647382    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647386    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647389    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647391    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647394    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647397    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647400    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647402    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647405    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647408    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647410    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647413    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.656336 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647415    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647418    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647422    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647425    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647429    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647431    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647434    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647438    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647441    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647444    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647446    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647449    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647451    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647454    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647456    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647459    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647462    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647465    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647468    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647470    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.656838 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647473    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647476    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647478    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647481    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647483    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647486    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647488    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647491    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647493    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647495    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647498    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647500    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647503    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647506    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647510    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647513    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647516    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647519    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647521    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647524    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.657358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647527    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647529    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647532    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647536    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647539    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647542    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647544    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647547    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647549    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647552    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647555    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647557    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647560    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647563    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647566    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647568    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647571    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647573    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.657862 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647576    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647578    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647581    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647583    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647586    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647589    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647591    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647594    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647598    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647601    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647604    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647607    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647609    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647612    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647615    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647617    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647620    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647623    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647625    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.658298 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647628    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647630    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647633    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647635    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647638    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647641    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647643    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647646    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647648    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647651    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647654    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647657    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647659    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647662    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647665    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647668    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.658744 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647670    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.647694    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647728    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647732    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647736    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647739    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647743    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647750    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647753    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647756    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647759    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647761    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647764    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647767    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647770    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.659316 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647772    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647775    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647778    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647781    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647783    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647786    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647802    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647806    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647809    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647811    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647814    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647817    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647820    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647823    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647825    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647828    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647830    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647833    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647836    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.659673 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647838    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647841    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647843    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647846    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647850    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647854    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647857    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647859    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647862    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647865    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647868    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647870    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647873    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647876    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647879    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647881    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647884    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647886    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647889    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647892    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.660183 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647895    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647897    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647900    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647902    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647905    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647908    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647911    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647913    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647915    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647918    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647920    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647923    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647926    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647928    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647931    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647933    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647940    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647945    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647947    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647950    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.660750 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647953    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647956    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647958    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647961    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647963    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647966    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647969    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647971    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647974    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647976    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647979    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647981    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647984    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647986    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647989    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647991    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647994    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647996    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.647999    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648002    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.661358 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648004    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648007    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648010    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648012    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648015    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648018    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648020    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648023    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648026    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648030    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648033    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648036    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648040    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.648043    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.653196    4528 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.653213    4528 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.653232    4528 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.653237    4528 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653299    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.661859 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653304    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653307    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653311    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653314    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653317    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653320    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653322    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653325    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653328    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653331    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653333    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653336    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653339    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653342    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653345    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653348    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653350    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653353    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653355    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653358    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.662306 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653361    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653363    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653366    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653369    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653371    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653375    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653378    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653385    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653389    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653391    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653394    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653397    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653399    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653402    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653405    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653407    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653410    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653412    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653415    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653418    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.662893 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653421    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653423    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653426    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653429    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653432    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653441    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653457    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653460    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653463    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653466    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653468    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653471    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653474    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653476    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653479    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653482    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653484    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653487    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653490    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653492    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.663347 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653495    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653497    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653500    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653502    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653505    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653508    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653510    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653513    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653516    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653518    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653521    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653523    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653526    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653528    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653532    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653536    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653539    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653542    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653546    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.663861 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653549    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653552    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653554    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653557    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653559    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653562    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653565    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653567    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653570    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653572    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653575    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653577    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653580    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653582    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653585    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653588    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653590    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653593    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653597    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.664335 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653601    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653604    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653607    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653609    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653612    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653615    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653617    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.653645    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653784    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653788    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653805    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653808    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653811    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653814    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653817    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.664856 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653821    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653825    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653828    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653831    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653834    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653836    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653839    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653841    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653844    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653846    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653849    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653852    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653854    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653857    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653859    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653862    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653864    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653867    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653870    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653872    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.665215 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653875    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653877    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653879    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653882    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653885    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653887    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653890    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653893    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653896    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653898    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653901    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653904    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653907    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653909    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653912    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653914    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653917    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653919    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653922    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.665692 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653925    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653927    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653930    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653932    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653935    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653937    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653940    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653942    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653945    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653947    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653950    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653953    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653955    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653957    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653960    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653963    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653965    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653968    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653971    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653973    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.666150 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653976    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653978    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653981    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653986    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653988    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653992    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653994    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653997    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.653999    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654002    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654005    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654007    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654009    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654012    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654015    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654017    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654020    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654022    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654025    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654027    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.666620 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654030    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654032    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654035    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654037    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654040    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654043    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654045    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654048    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654050    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654053    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654055    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654058    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654060    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654063    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654066    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654070    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654072    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654076    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654080    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.667094 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654083    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.654107    4528 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654138    4528 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654142    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654145    4528 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654147    4528 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654150    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654153    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654156    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654158    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654161    4528 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654165    4528 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654169    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654171    4528 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654174    4528 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 05:55:11.667549 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654176    4528 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654179    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654182    4528 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654184    4528 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654187    4528 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654190    4528 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654193    4528 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654197    4528 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654200    4528 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654202    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654205    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654207    4528 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654210    4528 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654214    4528 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654217    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654219    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654222    4528 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654231    4528 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654234    4528 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 05:55:11.667903 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654236    4528 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654239    4528 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654241    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654244    4528 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654247    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654249    4528 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654252    4528 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654255    4528 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654257    4528 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654260    4528 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654262    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654265    4528 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654267    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654270    4528 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654272    4528 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654276    4528 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654279    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654281    4528 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654284    4528 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 05:55:11.668365 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654286    4528 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654289    4528 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654291    4528 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654294    4528 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654296    4528 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654299    4528 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654301    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654305    4528 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654308    4528 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654311    4528 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654313    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654316    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654319    4528 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654321    4528 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654324    4528 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654326    4528 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654329    4528 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654331    4528 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654334    4528 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654338    4528 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 05:55:11.668961 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654340    4528 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654343    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654346    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654348    4528 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654351    4528 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654353    4528 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654356    4528 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654359    4528 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654361    4528 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654364    4528 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654366    4528 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654369    4528 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654371    4528 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654374    4528 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654376    4528 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654379    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654381    4528 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654384    4528 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654387    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654390    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 05:55:11.669723 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654393    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654396    4528 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654399    4528 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654401    4528 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654404    4528 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654407    4528 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654410    4528 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654412    4528 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654415    4528 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654417    4528 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654420    4528 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654423    4528 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654427    4528 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654429    4528 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:11.654432    4528 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.654487    4528 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655472    4528 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655522    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655701    4528 server.go:1014] "Starting client certificate rotation"
Nov 03 05:55:11.670454 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655827    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655878    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 05:49:31 +0000 UTC" deadline="2028-06-14 20:45:40.508453566 +0000 UTC"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.655934    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14150h50m28.852521628s"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.656182    4528 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.657246    4528 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.658651    4528 log.go:25] "Validated CRI v1 runtime API"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.660248    4528 log.go:25] "Validated CRI v1 image API"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.661014    4528 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 05:55:11.671129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.662602    4528 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a05087a3-1960-4a88-8686-4b09d87fc7f4:/dev/nvme0n1p3 c86b0c0d-73a6-4d06-8be4-41849362e17d:/dev/nvme0n1p4]
Nov 03 05:55:11.671489 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.662621    4528 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92/userdata/shm major:0 minor:241 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184/userdata/shm major:0 minor:208 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919/userdata/shm major:0 minor:224 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19e4e8ea-dd59-4bcb-924c-870fd4392bcf/volumes/kubernetes.io~projected/kube-api-access-sgx64:{mountpoint:/var/lib/kubelet/pods/19e4e8ea-dd59-4bcb-924c-870fd4392bcf/volumes/kubernetes.io~projected/kube-api-access-sgx64 major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/23536828-f0dc-4f42-8f23-988d8de6a262/volumes/kubernetes.io~projected/kube-api-access-98sp7:{mountpoint:/var/lib/kubelet/pods/23536828-f0dc-4f42-8f23-988d8de6a262/volumes/kubernetes.io~projected/kube-api-access-98sp7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2fc905f9-78f6-410f-8736-48ce0bced9d2/volumes/kubernetes.io~projected/kube-api-access-5dkk8:{mountpoint:/var/lib/kubelet/pods/2fc905f9-78f6-410f-8736-48ce0bced9d2/volumes/kubernetes.io~projected/kube-api-access-5dkk8 major:0 minor:222 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37062547-bd74-437f-a325-dbb4a09bffe1/volumes/kubernetes.io~projected/kube-api-access-j6wrz:{mountpoint:/var/lib/kubelet/pods/37062547-bd74-437f-a325-dbb4a09bffe1/volumes/kubernetes.io~projected/kube-api-access-j6wrz major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~projected/kube-api-access-vr868:{mountpoint:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~projected/kube-api-access-vr868 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6e5c002c-8b36-498b-aa36-c93aa0b3cb89/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/6e5c002c-8b36-498b-aa36-c93aa0b3cb89/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7c828422-e7da-4ee5-b748-8e8827dded89/volumes/kubernetes.io~projected/kube-api-access-6d2p4:{mountpoint:/var/lib/kubelet/pods/7c828422-e7da-4ee5-b748-8e8827dded89/volumes/kubernetes.io~projected/kube-api-access-6d2p4 major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~projected/kube-api-access-snfcp:{mountpoint:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~projected/kube-api-access-snfcp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/81d514bd-df1e-428c-ab18-66f841c88928/volumes/kubernetes.io~projected/kube-api-access-fbbhl:{mountpoint:/var/lib/kubelet/pods/81d514bd-df1e-428c-ab18-66f841c88928/volumes/kubernetes.io~projected/kube-api-access-fbbhl major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~projected/kube-api-access-vds6d:{mountpoint:/var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~projected/kube-api-access-vds6d major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/87a29b40-9e0d-4dde-bedb-c9423919e268/volumes/kubernetes.io~projected/kube-api-access-76kc7:{mountpoint:/var/lib/kubelet/pods/87a29b40-9e0d-4dde-bedb-c9423919e268/volumes/kubernetes.io~projected/kube-api-access-76kc7 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~projected/kube-api-access-9k5jk:{mountpoint:/var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~projected/kube-api-access-9k5jk major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~secret/signing-key major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bd8c881e-4142-4e06-9053-8cc113417d69/volumes/kubernetes.io~projected/kube-api-access-nb64g:{mountpoint:/var/lib/kubelet/pods/bd8c881e-4142-4e06-9053-8cc113417d69/volumes/kubernetes.io~projected/kube-api-access-nb64g major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c82bd6b6-edf0-4def-8b73-969955c6e014/volumes/kubernetes.io~projected/kube-api-access-5kwd7:{mountpoint:/var/lib/kubelet/pods/c82bd6b6-edf0-4def-8b73-969955c6e014/volumes/kubernetes.io~projected/kube-api-access-5kwd7 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cac9a7a9-12dc-4699-bf59-da9a169f261d/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cac9a7a9-12dc-4699-bf59-da9a169f261d/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc744642-5f3e-43f7-a565-4db72473b5aa/volumes/kubernetes.io~projected/kube-api-access-nrjg9:{mountpoint:/var/lib/kubelet/pods/cc744642-5f3e-43f7-a565-4db72473b5aa/volumes/kubernetes.io~projected/kube-api-access-nrjg9 major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ced67fea-5376-4c80-81cd-bdb12cccbd53/volumes/kubernetes.io~projected/kube-api-access-w2kgp:{mountpoint:/var/lib/kubelet/pods/ced67fea-5376-4c80-81cd-bdb12cccbd53/volumes/kubernetes.io~projected/kube-api-access-w2kgp major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b/volumes/kubernetes.io~projected/kube-api-access-6jdd5:{mountpoint:/var/lib/kubelet/pods/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b/volumes/kubernetes.io~projected/kube-api-access-6jdd5 major:0 minor:223 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/66386b8a3fd20f1aa0cf239b7ec2fd580428a3e63422ee76630aa7f963f56a51/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/7c7fa857ceacb1dbcf0bf8bcf887bec68954ddf5779c425098e0d9bde3a110af/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/90aed502936567663f2638ebfacb9e709f3dfb2fdda8ac96cdf789a0893d66bf/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d664f689aa85c212f2c4ba90b388ca392ca590ec80264871c8a8a47aaf0cbcb3/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/f8eb68264bf25e10aed429465d15247639929d3a9f7e9fde1f74d5d6943b81a9/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/742fb6df3ff826920ecb5a65b56483d18e51a7b19101ddfdd5b0c47cd84ea529/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/2416f115ed04cbc390cb9e6c4021a1b5bce76831370081bf2185c0fd680b9633/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-133:{mountpoint:/var/lib/containers/storage/overlay/26406318e8ae0d38b6acaa9b612e8435434370c6d07eec95a40a633d9aabe618/merged major:0 minor:133 fsType:overlay blockSize:0} overlay_0-135:{mountpoint:/var/lib/containers/storage/overlay/17b4860e783e8046ce6d7dbdec512ede7a24bf80ac3d02da6a7ae5fcb2f64817/merged major:0 minor:135 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/19bddd997074865bcd197b64ccc0a10f210b8cdccda6aacda6192abe93678d09/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/3f439a7803d87f75f948edb96f3864fa6cf8f47560c4ad7f2145f23685536909/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/53f41c0f14e2015ae7fa84ff9aab077f79aefa3e0ca3f7f159ea1f478ac4cb82/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/1c598446eaa3be05771f3f89c8305a5c02d0bda24e0201bbc8db255f4507d496/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/1ebe28e7634c20e317dc32212c37ccb29c482adee9869707b63f7237095d077f/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/768ef7520c4c0558ff96ca2118e559a4511c96e4298c3130f03058e09d689939/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e3df49d587918dd8068923936832804bba8e70982cc7dab0de3f202d446582b0/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/67526a2c8a528bfea9718e68bb9922c4cf762c919a119b7851839439176c824f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/95d219df7bb1a7267724a9bbda4865535e531ca83c38c3cb9d997218f6af2a58/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/143a79f8fde3fcdeb4bffb2c118167b277ab089889c7b3bd9f9a11bd6fb37f68/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/b39cd8ec3c7cf5369f2b6f761c1783d463be5b8883d327aa42719c87c23d1d70/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/0b38b63a27eab3d6e54eeb7f9fbec7bbd028ce2210078aa0b5e05df5c20cc343/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-212:{mountpoint:/var/lib/containers/storage/overlay/979aebcee397b3e55059a739f3ead3a5f4f0e3daac1295ceb9569d8a394ff230/merged major:0 minor:212 fsType:overlay blockSize:0} overlay_0-214:{mountpoint:/var/lib/containers/storage/overlay/8d227c8ab74d9be4b391dbcb82cd6181e1158a4d06f2324369dbd3a1f0180e0f/merged major:0 minor:214 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/d94fe89801f9e65001787daa2596c43c80843093740fcf304d280fdb9d563e95/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/eb52c7477db7a55b0bec20187d6c4388fa33736f61ce49e8f50f4d8a4754a544/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/be4565ba08b40487247b78c2d1cba41fb78e8b446e2ccb612fe1f26773fc6ef2/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-238:{mountpoint:/var/lib/containers/storage/overlay/62ed22e53d50873cb09a107e0e056c453f459dbeef9ae4d3fb583b653e1ac352/merged major:0 minor:238 fsType:overlay blockSize:0} overlay_0-243:{mountpoint:/var/lib/containers/storage/overlay/691992e1df1edbb7f51eb3f9598f2ac7b90845115e2b2abda7bcf80becf0f76e/merged major:0 minor:243 fsType:overlay blockSize:0} overlay_0-245:{mountpoint:/var/lib/containers/storage/overlay/05cc567dd86d6de93d3536cf1bea5e56b0821b6f0c1206ef6c2e2208e220b9d2/merged major:0 minor:245 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/2de72d5194314ea75c6b0269dd84f23d98fd551514aeba423bdeb8992aba516b/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/023f45df4ba02101dd9d701bfdd18790ca608bf4a3d67d7ded88925b41ce6089/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/6ab3620fce101308d015ef810d42049b32914b63113154a6c115ab54c509920e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/82cdc633dccfeea94f6ac0fc1e940c325d0f6e4e6e7d9517ee9abea724c336c2/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/a92938306bf44c34a09150f3e51efab5580054384ab4e157ef44f2a22278707a/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/affee558ee82c9f78c89b6b39075b317cb4343f87349ca84d2000c5ba7be8e50/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/8bd0fd5bd85b0b942a3462c44cd05297fcca9dc29acd014ca25990b9d1650dcd/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/e691dc333e9c6dcd63de57d26dc2809c249bdf38bb15760a13553ae630ac5550/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/ae609985278c12566a614e5c3d661016bc3c44a101708eec9da39586a7a24cce/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/e4f386f08f9cb169f4c3fa067d746cc1a7b88229e151c013f75cb9be4dd83657/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/34e95bca2c5a3986b9b00979c347b89436cc93baad90d824d6d02c6d3028e767/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-99:{mountpoint:/var/lib/containers/storage/overlay/906b7492274d585e18ddeed5ff1509c4c1628d7e2f55522da61ed5f707d10ea0/merged major:0 minor:99 fsType:overlay blockSize:0}]
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.670719    4528 manager.go:217] Machine: {Timestamp:2026-11-03 05:55:11.669668858 +0000 UTC m=+0.081065926 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3120587 MemoryCapacity:33164337152 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec292e8f39fc025f09a4753cbc6b0655 SystemUUID:ec292e8f-39fc-025f-09a4-753cbc6b0655 BootID:5c55df60-70a7-45c5-8229-21f10a76eceb Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-135 DeviceMajor:0 DeviceMinor:135 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2fc905f9-78f6-410f-8736-48ce0bced9d2/volumes/kubernetes.io~projected/kube-api-access-5dkk8 DeviceMajor:0 DeviceMinor:222 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/87a29b40-9e0d-4dde-bedb-c9423919e268/volumes/kubernetes.io~projected/kube-api-access-76kc7 DeviceMajor:0 DeviceMinor:67 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7c828422-e7da-4ee5-b748-8e8827dded89/volumes/kubernetes.io~projected/kube-api-access-6d2p4 DeviceMajor:0 DeviceMinor:197 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/632615d10bb959c1b4157ff5af1b70ab7e2eab847ba43254c0202d08fe058c21/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/81d514bd-df1e-428c-ab18-66f841c88928/volumes/kubernetes.io~projected/kube-api-access-fbbhl DeviceMajor:0 DeviceMinor:65 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~projected/kube-api-access-vr868 DeviceMajor:0 DeviceMinor:66 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-133 DeviceMajor:0 DeviceMinor:133 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cbf685f9df1ab1edc22418f30b01c453e1d72b34bbb5eff6439f2f1dd12ee8f/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-212 DeviceMajor:0 DeviceMinor:212 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19e4e8ea-dd59-4bcb-924c-870fd4392bcf/volumes/kubernetes.io~projected/kube-api-access-sgx64 DeviceMajor:0 DeviceMinor:226 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-245 DeviceMajor:0 DeviceMinor:245 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c82bd6b6-edf0-4def-8b73-969955c6e014/volumes/kubernetes.io~projected/kube-api-access-5kwd7 DeviceMajor:0 DeviceMinor:60 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~projected/kube-api-access-snfcp DeviceMajor:0 DeviceMinor:61 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/23536828-f0dc-4f42-8f23-988d8de6a262/volumes/kubernetes.io~projected/kube-api-access-98sp7 DeviceMajor:0 DeviceMinor:64 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-243 DeviceMajor:0 DeviceMinor:243 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bd8c881e-4142-4e06-9053-8cc113417d69/volumes/kubernetes.io~projected/kube-api-access-nb64g DeviceMajor:0 DeviceMinor:59 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/54b1954459353493c146015cb0d551c7af1171fcaca58baf29d9d8d2e84158c8/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/799d61da59b011b54d6c3405f5cd47256fc41eb64b5dfe3f5e6fbca869d16184/userdata/shm DeviceMajor:0 DeviceMinor:208 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9a7d585ba94859230b822b7ed4fc031fd41ae8c8651739caafc7cde7c878e26e/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e46684ca41c6c3cf35e8e99f6517b19fd3fa10c1ab4ac1e38d97c20bff04344e/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ced67fea-5376-4c80-81cd-bdb12cccbd53/volumes/kubernetes.io~projected/kube-api-access-w2kgp DeviceMajor:0 DeviceMinor:62 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e03d8944765a26d031171eb60fd687c57a43e05a37142a6111d8f291ccc5e6c1/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-214 DeviceMajor:0 DeviceMinor:214 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/42930b20fa2910ebecdf6e1907954bd1b1aade67493cf744c500903293a27adb/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6e5c002c-8b36-498b-aa36-c93aa0b3cb89/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:206 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b/volumes/kubernetes.io~projected/kube-api-access-6jdd5 DeviceMajor:0 DeviceMinor:223 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/37062547-bd74-437f-a325-dbb4a09bffe1/volumes/kubernetes.io~projected/kube-api-access-j6wrz DeviceMajor:0 DeviceMinor:63 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-99 DeviceMajor:0 DeviceMinor:99 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919/userdata/shm DeviceMajor:0 DeviceMinor:224 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~projected/kube-api-access-9k5jk DeviceMajor:0 DeviceMinor:240 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0ad7a46a1086f41ed97d216b5fe0d4b76f9c875cae304ecc0c873147e7824b21/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/24f7448336799fb9d785c12887ccf978070e2338614182d202ffbe1ba47511c9/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7f2b850b-029d-42d2-aeb0-3856d68104f8/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/87999404-7a09-46b3-8d41-65407bb93863/volumes/kubernetes.io~projected/kube-api-access-vds6d DeviceMajor:0 DeviceMinor:201 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/169d9b1560852c758bc314970ba70639cb394944f5090d4731d6c365625eea92/userdata/shm DeviceMajor:0 DeviceMinor:241 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed33c44bb452e57459b9a3852aa2253747084eb83fc4b92b4a4cac258c03a980/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:233 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/4d92dbe9-16b1-4412-a14d-a2e7367d169a/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc744642-5f3e-43f7-a565-4db72473b5aa/volumes/kubernetes.io~projected/kube-api-access-nrjg9 DeviceMajor:0 DeviceMinor:207 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cac9a7a9-12dc-4699-bf59-da9a169f261d/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985737728 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b3af3745a6ac1231abe8ec877a75015002fef2a1c73b5f7915759bba6949e35b/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-238 DeviceMajor:0 DeviceMinor:238 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:169d9b1560852c7 MacAddress:16:96:a8:26:cb:94 Speed:10000 Mtu:8901} {Name:3cbf685f9df1ab1 MacAddress:42:a7:fd:bf:94:1a Speed:10000 Mtu:8901} {Name:632615d10bb959c MacAddress:52:8b:bd:57:1f:ef Speed:10000 Mtu:8901} {Name:799d61da59b011b MacAddress:46:ed:12:5f:48:49 Speed:10000 Mtu:8901} {Name:b33152a5599f9da MacAddress:ba:b8:eb:20:3d:54 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:d5:1d:c4:03 Speed:0 Mtu:9001} {Name:br-int MacAddress:fe:a4:85:20:b2:d0 Speed:0 Mtu:8901} {Name:ed33c44bb452e57 MacAddress:f2:e0:15:93:b8:67 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:d5:1d:c4:03 Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:02:86:21:4a:87:85 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:d6:2e:d6:5e:6a:3e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164337152 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671028    4528 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671101    4528 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671451    4528 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671486    4528 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-132-233.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671666    4528 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671676    4528 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671689    4528 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 05:55:11.671725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671707    4528 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.671844    4528 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672048    4528 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672174    4528 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672223    4528 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672262    4528 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672276    4528 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672290    4528 kubelet.go:405] "Adding apiserver pod source"
Nov 03 05:55:11.672402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672301    4528 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 05:55:11.673012 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.672996    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 05:55:11.673012 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.673013    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 05:55:11.673529 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.673505    4528 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 05:55:11.700493 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700477    4528 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 05:55:11.700493 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700492    4528 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 05:55:11.700655 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700608    4528 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 05:55:11.700825 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700813    4528 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 05:55:11.700856 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700843    4528 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 05:55:11.700961 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700951    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 05:55:11.701001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700971    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 05:55:11.701001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700981    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 05:55:11.701001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700989    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 05:55:11.701001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.700996    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 05:55:11.701001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701003    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701009    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701015    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701021    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701032    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701045    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701054    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701079    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 05:55:11.701178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.701084    4528 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 05:55:11.704311 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.704297    4528 server.go:1263] "Started kubelet"
Nov 03 05:55:11.705076 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.704410    4528 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 05:55:11.705076 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.704475    4528 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 05:55:11.705076 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.704574    4528 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 05:55:11.705076 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.705020    4528 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 05:55:11.704931 ip-10-0-132-233 systemd[1]: Started Kubernetes Kubelet.
Nov 03 05:55:11.705927 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.705911    4528 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 05:55:11.710810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.710777    4528 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 05:55:11.710891 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.710866    4528 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 05:55:11.710956 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.710897    4528 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-132-233.ec2.internal\" not found"
Nov 03 05:55:11.711450 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.711422    4528 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 05:55:11.711450 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.711438    4528 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 05:55:11.711581 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.711531    4528 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 05:49:33 +0000 UTC" deadline="2028-07-23 10:55:52.624252706 +0000 UTC"
Nov 03 05:55:11.711632 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.711582    4528 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15077h0m40.912674163s"
Nov 03 05:55:11.711697 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.711678    4528 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 05:55:11.712715 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.712697    4528 factory.go:55] Registering systemd factory
Nov 03 05:55:11.712715 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.712716    4528 factory.go:223] Registration of the systemd container factory successfully
Nov 03 05:55:11.713182 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713140    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d514bd-df1e-428c-ab18-66f841c88928" volumeName="kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713180    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87999404-7a09-46b3-8d41-65407bb93863" volumeName="kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713198    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37062547-bd74-437f-a325-dbb4a09bffe1" volumeName="kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713210    4528 factory.go:153] Registering CRI-O factory
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713211    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d92dbe9-16b1-4412-a14d-a2e7367d169a" volumeName="kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713230    4528 factory.go:223] Registration of the crio container factory successfully
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713235    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2b850b-029d-42d2-aeb0-3856d68104f8" volumeName="kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713247    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ced67fea-5376-4c80-81cd-bdb12cccbd53" volumeName="kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713260    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23536828-f0dc-4f42-8f23-988d8de6a262" volumeName="kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713275    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37062547-bd74-437f-a325-dbb4a09bffe1" volumeName="kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 05:55:11.713277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713280    4528 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713290    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2b850b-029d-42d2-aeb0-3856d68104f8" volumeName="kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713298    4528 factory.go:103] Registering Raw factory
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713301    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87999404-7a09-46b3-8d41-65407bb93863" volumeName="kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713313    4528 manager.go:1199] Started watching for new ooms in manager
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713327    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8c881e-4142-4e06-9053-8cc113417d69" volumeName="kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713340    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cac9a7a9-12dc-4699-bf59-da9a169f261d" volumeName="kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713355    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2b850b-029d-42d2-aeb0-3856d68104f8" volumeName="kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713370    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62" volumeName="kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713388    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62" volumeName="kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713405    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62" volumeName="kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713421    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc905f9-78f6-410f-8736-48ce0bced9d2" volumeName="kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713446    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37062547-bd74-437f-a325-dbb4a09bffe1" volumeName="kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713457    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c828422-e7da-4ee5-b748-8e8827dded89" volumeName="kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713471    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c828422-e7da-4ee5-b748-8e8827dded89" volumeName="kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713496    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2b850b-029d-42d2-aeb0-3856d68104f8" volumeName="kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713508    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c82bd6b6-edf0-4def-8b73-969955c6e014" volumeName="kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713518    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c82bd6b6-edf0-4def-8b73-969955c6e014" volumeName="kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713530    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19e4e8ea-dd59-4bcb-924c-870fd4392bcf" volumeName="kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713542    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d92dbe9-16b1-4412-a14d-a2e7367d169a" volumeName="kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713580    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6e5c002c-8b36-498b-aa36-c93aa0b3cb89" volumeName="kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713591    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7f2b850b-029d-42d2-aeb0-3856d68104f8" volumeName="kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713601    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="81d514bd-df1e-428c-ab18-66f841c88928" volumeName="kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713620    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bd8c881e-4142-4e06-9053-8cc113417d69" volumeName="kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713633    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cac9a7a9-12dc-4699-bf59-da9a169f261d" volumeName="kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713644    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc744642-5f3e-43f7-a565-4db72473b5aa" volumeName="kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713657    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2fc905f9-78f6-410f-8736-48ce0bced9d2" volumeName="kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713670    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d92dbe9-16b1-4412-a14d-a2e7367d169a" volumeName="kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713693    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7c828422-e7da-4ee5-b748-8e8827dded89" volumeName="kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713705    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c82bd6b6-edf0-4def-8b73-969955c6e014" volumeName="kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713712    4528 manager.go:319] Starting recovery of all containers
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713716    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" volumeName="kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713727    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" volumeName="kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713740    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" volumeName="kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713754    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" volumeName="kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713766    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="23536828-f0dc-4f42-8f23-988d8de6a262" volumeName="kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca" seLinuxMountContext=""
Nov 03 05:55:11.713760 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713777    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37062547-bd74-437f-a325-dbb4a09bffe1" volumeName="kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz" seLinuxMountContext=""
Nov 03 05:55:11.715396 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713805    4528 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="87a29b40-9e0d-4dde-bedb-c9423919e268" volumeName="kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7" seLinuxMountContext=""
Nov 03 05:55:11.715396 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713818    4528 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 05:55:11.715396 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.713825    4528 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 05:55:11.730377 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.730349    4528 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:55:11.739382 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.739351    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 05:55:11.740543 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.740520    4528 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 05:55:11.740543 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.740544    4528 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 05:55:11.740684 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.740563    4528 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 05:55:11.740684 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.740606    4528 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 05:55:11.743487 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.743431    4528 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:55:11.743612 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.743597    4528 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:55:11.743716 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.743671    4528 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 05:55:11.752952 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752907    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="647a099ed233fdc4bfce94edfb3706c855d067b9ac452a38721134a948a09e8b" exitCode=0
Nov 03 05:55:11.753029 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752952    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="efa623b1aee0c2f4497d16e96419228b9cb7923a8a40f4a3763fbea8990c734b" exitCode=0
Nov 03 05:55:11.753029 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752963    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="26ecdcbca36c65c74c6eebb49c2bd8c0ed58eb4fbf6432600ab705b314006c89" exitCode=0
Nov 03 05:55:11.753029 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752972    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="219c096ec1354322c072ebfbd276688a7af6d8778dd4d3fb541fac7b0772d575" exitCode=0
Nov 03 05:55:11.753029 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752981    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="30c2b02d20f93680fe7cbc7ea15c2f02ef164999f704544bce49edcee1c02f65" exitCode=0
Nov 03 05:55:11.753029 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.752990    4528 generic.go:356] "Generic (PLEG): container finished" podID="37062547-bd74-437f-a325-dbb4a09bffe1" containerID="70efa738cd19217725558195123b5dfd6924c9a55bc7e458de9975dace0f65e3" exitCode=0
Nov 03 05:55:11.757605 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.757583    4528 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 05:55:11.757924 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.757900    4528 generic.go:356] "Generic (PLEG): container finished" podID="9bc364535d6718828aba0cd8493dd684" containerID="e4e8886420e10c5a4d141729a9f2e80f2eda333c21b68dc7768627ac06ad1984" exitCode=0
Nov 03 05:55:11.785385 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.785324    4528 manager.go:324] Recovery completed
Nov 03 05:55:11.800150 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800130    4528 cpu_manager.go:226] "Starting" policy="none"
Nov 03 05:55:11.800150 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800142    4528 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800157    4528 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800287    4528 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800299    4528 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800315    4528 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800320    4528 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 05:55:11.800326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800325    4528 policy_none.go:50] "Start"
Nov 03 05:55:11.800521 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800333    4528 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 05:55:11.800521 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800342    4528 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 05:55:11.800521 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800457    4528 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 05:55:11.800521 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800465    4528 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 05:55:11.800521 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.800471    4528 policy_none.go:44] "Start"
Nov 03 05:55:11.804862 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.804357    4528 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 05:55:11.809896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.809882    4528 manager.go:341] "Starting Device Plugin manager"
Nov 03 05:55:11.809991 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.809905    4528 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 05:55:11.809991 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.809913    4528 server.go:87] "Starting device plugin registration server"
Nov 03 05:55:11.810217 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810203    4528 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 05:55:11.810274 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810217    4528 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 05:55:11.810330 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810306    4528 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 05:55:11.810431 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810397    4528 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 05:55:11.810508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810437    4528 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 05:55:11.810508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810443    4528 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 05:55:11.810608 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.810488    4528 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T05:55:11.810428215Z","UUID":"e8043064-9c1c-4def-8deb-14216d835944","Handler":null,"Name":"","Endpoint":""}
Nov 03 05:55:11.811780 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.811764    4528 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 05:55:11.811929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.811912    4528 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 05:55:11.811987 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.811944    4528 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 05:55:11.840744 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.840706    4528 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"]
Nov 03 05:55:11.841178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841148    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerStarted","Data":"b37a6db27cae845a576873eb6d4224ed11a29f322c45de6dcf6bcb45f9adc6ba"}
Nov 03 05:55:11.841178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841182    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerDied","Data":"e4e8886420e10c5a4d141729a9f2e80f2eda333c21b68dc7768627ac06ad1984"}
Nov 03 05:55:11.841321 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841196    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal" event={"ID":"9bc364535d6718828aba0cd8493dd684","Type":"ContainerStarted","Data":"0ed67491a094fb46b6661bc9e8692c61a25f6d3de22392902399b5bdc57b810d"}
Nov 03 05:55:11.841321 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841227    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal" event={"ID":"1b9c97e9d8c2351046139958cd95120b","Type":"ContainerStarted","Data":"021f03637d7c470d861b14c404f7c15dae50047214fdee32179ffd615f4533a9"}
Nov 03 05:55:11.841321 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841238    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal" event={"ID":"1b9c97e9d8c2351046139958cd95120b","Type":"ContainerStarted","Data":"5afecdec5880404d2b04dcb4d84bf0c5f1bdab17e309cf10f8108e082fa19e73"}
Nov 03 05:55:11.841321 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841252    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.841516 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.841344    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.865643 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.865626    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:55:11.865711 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.865668    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.866090 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.866077    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:55:11.866142 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:11.866112    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.910657 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.910632    4528 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 05:55:11.911676 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.911663    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 05:55:11.911750 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.911686    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 05:55:11.911750 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.911695    4528 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-132-233.ec2.internal" event="NodeHasSufficientPID"
Nov 03 05:55:11.911750 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.911722    4528 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.945224 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.945199    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.945328 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.945227    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.945328 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.945246    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.945639 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.945621    4528 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:55:11.945691 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:11.945682    4528 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046074 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.045975    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046074 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.046029    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046074 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.046082    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046289 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.046089    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1b9c97e9d8c2351046139958cd95120b-config\") pod \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" (UID: \"1b9c97e9d8c2351046139958cd95120b\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046289 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.046137    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.046289 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.046173    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc364535d6718828aba0cd8493dd684-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" (UID: \"9bc364535d6718828aba0cd8493dd684\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.672761 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.672718    4528 apiserver.go:52] "Watching apiserver"
Nov 03 05:55:12.722858 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.722825    4528 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 05:55:12.723327 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.723301    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-bws8t","openshift-service-ca/service-ca-5fffc97bdf-8gmvq","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal","openshift-dns/node-resolver-hvprb","openshift-multus/multus-additional-cni-plugins-lgrkn","openshift-ovn-kubernetes/ovnkube-node-hthhm","openshift-insights/insights-runtime-extractor-5v8g7","openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4","openshift-network-operator/iptables-alerter-lgjdg","kube-system/konnectivity-agent-xbkcj","kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal","kube-system/global-pull-secret-syncer-bxj2x","openshift-insights/periodic-gathering-z2tsc-b8vnj","openshift-network-diagnostics/network-check-target-g5slj","openshift-dns/dns-default-p8jsb","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4","openshift-cluster-node-tuning-operator/tuned-48zrq","openshift-ingress-canary/ingress-canary-hxvcn","openshift-multus/multus-2zxms","openshift-multus/network-metrics-daemon-glzh2"]
Nov 03 05:55:12.723923 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.723901    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:12.727538 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.727521    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.728869 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.728849    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.742717 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.742692    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 05:55:12.742991 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.742975    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 05:55:12.743269 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743254    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.743269 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743255    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.743613 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743598    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 05:55:12.743713 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743696    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 05:55:12.743770 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743758    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 05:55:12.743812 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743765    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 05:55:12.743849 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743833    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.744014 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.743998    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-mlps4\""
Nov 03 05:55:12.744066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744014    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.744066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744024    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-fkxng\""
Nov 03 05:55:12.744066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744036    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 05:55:12.744066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744043    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 05:55:12.744066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744061    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.744250 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744068    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-rwdxr\""
Nov 03 05:55:12.744250 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744108    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-nphqj\""
Nov 03 05:55:12.744250 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744153    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-gfnmx\""
Nov 03 05:55:12.744250 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744204    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zq86f\""
Nov 03 05:55:12.744250 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744224    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.744415 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.744315    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 05:55:12.751473 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.751458    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-z796t\""
Nov 03 05:55:12.751924 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.751910    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xvf66\""
Nov 03 05:55:12.752079 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752065    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-8hrbq\""
Nov 03 05:55:12.752141 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752083    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 05:55:12.752141 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752095    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 05:55:12.752141 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752104    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j9vz5\""
Nov 03 05:55:12.752141 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752066    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-75slc\""
Nov 03 05:55:12.752347 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752331    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 05:55:12.752380 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752339    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-krwzb\""
Nov 03 05:55:12.752380 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.752371    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-xj6m8\""
Nov 03 05:55:12.753356 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.753342    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-jcdvc\""
Nov 03 05:55:12.753423 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.753411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-tn6jl\""
Nov 03 05:55:12.753471 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.753444    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nvbtr\""
Nov 03 05:55:12.754752 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.754738    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 05:55:12.754839 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.754776    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 05:55:12.754839 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.754775    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Nov 03 05:55:12.754917 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.754839    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-hwk29\""
Nov 03 05:55:12.754917 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.754843    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-7glbz\""
Nov 03 05:55:12.762305 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.762290    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 05:55:12.762363 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.762315    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.762770 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.762753    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.762899 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.762883    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763153 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763139    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 05:55:12.763215 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763183    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763215 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763199    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763294 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763214    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763339 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763316    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 05:55:12.763339 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763328    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763331    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763342    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763359    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763361    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763374    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763391    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763397    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763399    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763408    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763415    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 03 05:55:12.763430 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763430    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763936 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763451    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 05:55:12.763936 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763765    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763936 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763809    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 05:55:12.763936 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.763910    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Nov 03 05:55:12.764350 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.764334    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 05:55:12.765259 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.765242    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 03 05:55:12.781340 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.781316    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.781476 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.781454    4528 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.805106 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.805082    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:55:12.805185 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.805130    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-132-233.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.808457 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.808444    4528 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 05:55:12.808507 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.808474    4528 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-132-233.ec2.internal"
Nov 03 05:55:12.812996 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.812981    4528 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 05:55:12.849200 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vds6d\" (UniqueName: \"kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:12.849200 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:12.849383 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849275    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:55:12.849383 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.849383 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849330    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.849383 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.849625 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849452    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.849625 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.849625 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849548    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.849625 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849576    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849643    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849675    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849709    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849739    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849768    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.849838 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849818    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849845    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849847    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cac9a7a9-12dc-4699-bf59-da9a169f261d-agent-certs\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849870    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849875    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-daemon-config\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c82bd6b6-edf0-4def-8b73-969955c6e014-cni-binary-copy\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849895    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.849969    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850012    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850044    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.850115 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850094    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850639 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850308    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.850639 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850413    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovn-node-metrics-cert\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850639 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850478    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.850739 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850167    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850739 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850685    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850739 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850712    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.850739 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850707    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xbkcj_cac9a7a9-12dc-4699-bf59-da9a169f261d/konnectivity-agent/0.log" limit=1024
Nov 03 05:55:12.850925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850739    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.850925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850756    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.850925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850780    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.850925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850852    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.850925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850873    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-tuned\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850925    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.850975    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851008    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4d92dbe9-16b1-4412-a14d-a2e7367d169a-tmp\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851058    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851101    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr868\" (UniqueName: \"kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851108    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-original-pull-secret\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/23536828-f0dc-4f42-8f23-988d8de6a262-serviceca\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851264    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851294    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851311    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbbhl\" (UniqueName: \"kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.851391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851390    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.851636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/2fc905f9-78f6-410f-8736-48ce0bced9d2-data-volume\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.851636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851580    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.851636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851624    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.851636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-key\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851650    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snfcp\" (UniqueName: \"kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851694    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851724    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851747    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851804    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.851847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851838    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851868    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgx64\" (UniqueName: \"kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64\") pod \"migrator-66df5d496b-hhsb4\" (UID: \"19e4e8ea-dd59-4bcb-924c-870fd4392bcf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851909    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-env-overrides\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.851961    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852056    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852077    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.852136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852127    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-signing-cabundle\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852145    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852256    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852321    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852346    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.852394 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852376    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852466    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sp7\" (UniqueName: \"kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852470    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76kc7\" (UniqueName: \"kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852585    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/87999404-7a09-46b3-8d41-65407bb93863-cert\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852595    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb64g\" (UniqueName: \"kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.852660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852645    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852700    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852739    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c828422-e7da-4ee5-b748-8e8827dded89-tmp-dir\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852745    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852817    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852858    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852884    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852912    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852953    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.853032 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.852996    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853024    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k5jk\" (UniqueName: \"kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853066    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853091    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-binary-copy\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853122    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853150    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853188    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853201    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/37062547-bd74-437f-a325-dbb4a09bffe1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853238    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2kgp\" (UniqueName: \"kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81d514bd-df1e-428c-ab18-66f841c88928-tmp-dir\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.853409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853382    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d2p4\" (UniqueName: \"kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853476    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853552    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7c828422-e7da-4ee5-b748-8e8827dded89-config-volume\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853564    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853589    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853617    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853737    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853762    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-config\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853778    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853821    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853848    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.853896 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853875    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853910    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7f2b850b-029d-42d2-aeb0-3856d68104f8-ovnkube-script-lib\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853912    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853957    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6wrz\" (UniqueName: \"kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853975    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/bd8c881e-4142-4e06-9053-8cc113417d69-iptables-alerter-script\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.853987    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dkk8\" (UniqueName: \"kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.854008    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cac9a7a9-12dc-4699-bf59-da9a169f261d-konnectivity-ca\") pod \"konnectivity-agent-xbkcj\" (UID: \"cac9a7a9-12dc-4699-bf59-da9a169f261d\") " pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.854021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.854231 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.854047    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.876903 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.876881    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgx64\" (UniqueName: \"kubernetes.io/projected/19e4e8ea-dd59-4bcb-924c-870fd4392bcf-kube-api-access-sgx64\") pod \"migrator-66df5d496b-hhsb4\" (UID: \"19e4e8ea-dd59-4bcb-924c-870fd4392bcf\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-hhsb4"
Nov 03 05:55:12.876951 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.876884    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"periodic-gathering-z2tsc-b8vnj\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") " pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:55:12.880552 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.880535    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vds6d\" (UniqueName: \"kubernetes.io/projected/87999404-7a09-46b3-8d41-65407bb93863-kube-api-access-vds6d\") pod \"ingress-canary-hxvcn\" (UID: \"87999404-7a09-46b3-8d41-65407bb93863\") " pod="openshift-ingress-canary/ingress-canary-hxvcn"
Nov 03 05:55:12.881279 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.881262    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snfcp\" (UniqueName: \"kubernetes.io/projected/7f2b850b-029d-42d2-aeb0-3856d68104f8-kube-api-access-snfcp\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.882451 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.882437    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbbhl\" (UniqueName: \"kubernetes.io/projected/81d514bd-df1e-428c-ab18-66f841c88928-kube-api-access-fbbhl\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.882735 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.882721    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjg9\" (UniqueName: \"kubernetes.io/projected/cc744642-5f3e-43f7-a565-4db72473b5aa-kube-api-access-nrjg9\") pod \"network-check-target-g5slj\" (UID: \"cc744642-5f3e-43f7-a565-4db72473b5aa\") " pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:12.886137 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.886121    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kwd7\" (UniqueName: \"kubernetes.io/projected/c82bd6b6-edf0-4def-8b73-969955c6e014-kube-api-access-5kwd7\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.890758 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.890732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6wrz\" (UniqueName: \"kubernetes.io/projected/37062547-bd74-437f-a325-dbb4a09bffe1-kube-api-access-j6wrz\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.890877 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.890741    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76kc7\" (UniqueName: \"kubernetes.io/projected/87a29b40-9e0d-4dde-bedb-c9423919e268-kube-api-access-76kc7\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:12.890976 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.890879    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dkk8\" (UniqueName: \"kubernetes.io/projected/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-api-access-5dkk8\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.891152 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.891131    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb64g\" (UniqueName: \"kubernetes.io/projected/bd8c881e-4142-4e06-9053-8cc113417d69-kube-api-access-nb64g\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.891968 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.891947    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k5jk\" (UniqueName: \"kubernetes.io/projected/a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62-kube-api-access-9k5jk\") pod \"service-ca-5fffc97bdf-8gmvq\" (UID: \"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62\") " pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq"
Nov 03 05:55:12.892171 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.892154    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sp7\" (UniqueName: \"kubernetes.io/projected/23536828-f0dc-4f42-8f23-988d8de6a262-kube-api-access-98sp7\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.899998 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.899979    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2kgp\" (UniqueName: \"kubernetes.io/projected/ced67fea-5376-4c80-81cd-bdb12cccbd53-kube-api-access-w2kgp\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.899998 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.899988    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d2p4\" (UniqueName: \"kubernetes.io/projected/7c828422-e7da-4ee5-b748-8e8827dded89-kube-api-access-6d2p4\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.902586 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.902569    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr868\" (UniqueName: \"kubernetes.io/projected/4d92dbe9-16b1-4412-a14d-a2e7367d169a-kube-api-access-vr868\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.955066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.954987    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955020    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955040    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.955066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955054    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955094    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955109    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955114    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955152    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-bin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955115    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955150    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955123    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-system-cni-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-socket-dir-parent\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-multus-certs\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955203    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955226    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-hostroot\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955260    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955306    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955337    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-var-lib-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955346    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-etc-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-systemd-units\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-log-socket\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.955346 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955357    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-bin\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-netns\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955439    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/81d514bd-df1e-428c-ab18-66f841c88928-hosts-file\") pod \"node-resolver-hvprb\" (UID: \"81d514bd-df1e-428c-ab18-66f841c88928\") " pod="openshift-dns/node-resolver-hvprb"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955451    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955487    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955492    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-registration-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955501    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.955508    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-lib-modules\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955520    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955544    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955558    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysconfig\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955580    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955583    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-device-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.955596    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:13.455573567 +0000 UTC m=+1.866970642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.955604    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-etc-kubernetes\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955639    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.955670    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:13.455651731 +0000 UTC m=+1.867048799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955671    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955606    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-cnibin\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955688    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955681    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/23536828-f0dc-4f42-8f23-988d8de6a262-host\") pod \"node-ca-bws8t\" (UID: \"23536828-f0dc-4f42-8f23-988d8de6a262\") " pod="openshift-image-registry/node-ca-bws8t"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955712    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955728    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955742    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-kubelet\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955831    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955832    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-cni-multus\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-run-k8s-cni-cncf-io\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955844    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-dbus\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955866    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-openvswitch\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955884    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955906    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955909    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-modprobe-d\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955926    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955930    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-ovn\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-node-log\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955950    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955957    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-socket-dir\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955973    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-cni-netd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956005    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-host\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.955985    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956036    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956056    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-systemd\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956137    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-sys\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956140    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956096    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-etc-selinux\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956168    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-sysctl-conf\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956177    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.956206    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956261    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956268    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-host-var-lib-kubelet\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956208    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-var-lib-kubelet\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:12.956278    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:13.456263994 +0000 UTC m=+1.867661054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:12.956277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956312    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956338    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956360    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956382    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956387    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956391    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-run-systemd\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956417    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-run-netns\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7f2b850b-029d-42d2-aeb0-3856d68104f8-host-slash\") pod \"ovnkube-node-hthhm\" (UID: \"7f2b850b-029d-42d2-aeb0-3856d68104f8\") " pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956426    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956458    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-etc-kubernetes\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956482    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/bd8c881e-4142-4e06-9053-8cc113417d69-host-slash\") pod \"iptables-alerter-lgjdg\" (UID: \"bd8c881e-4142-4e06-9053-8cc113417d69\") " pod="openshift-network-operator/iptables-alerter-lgjdg"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956459    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4d92dbe9-16b1-4412-a14d-a2e7367d169a-run\") pod \"tuned-48zrq\" (UID: \"4d92dbe9-16b1-4412-a14d-a2e7367d169a\") " pod="openshift-cluster-node-tuning-operator/tuned-48zrq"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956482    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956522    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/ced67fea-5376-4c80-81cd-bdb12cccbd53-sys-fs\") pod \"aws-ebs-csi-driver-node-6txm4\" (UID: \"ced67fea-5376-4c80-81cd-bdb12cccbd53\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6txm4"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956555    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956581    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956594    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/2fc905f9-78f6-410f-8736-48ce0bced9d2-crio-socket\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956627    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956650    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956672    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-multus-conf-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956707    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/37062547-bd74-437f-a325-dbb4a09bffe1-cnibin\") pod \"multus-additional-cni-plugins-lgrkn\" (UID: \"37062547-bd74-437f-a325-dbb4a09bffe1\") " pod="openshift-multus/multus-additional-cni-plugins-lgrkn"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956728    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6e5c002c-8b36-498b-aa36-c93aa0b3cb89-kubelet-config\") pod \"global-pull-secret-syncer-bxj2x\" (UID: \"6e5c002c-8b36-498b-aa36-c93aa0b3cb89\") " pod="kube-system/global-pull-secret-syncer-bxj2x"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.956849    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c82bd6b6-edf0-4def-8b73-969955c6e014-system-cni-dir\") pod \"multus-2zxms\" (UID: \"c82bd6b6-edf0-4def-8b73-969955c6e014\") " pod="openshift-multus/multus-2zxms"
Nov 03 05:55:12.958288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:12.957129    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/2fc905f9-78f6-410f-8736-48ce0bced9d2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:13.029711 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.029691    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 05:55:13.227929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.227851    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:13.341643 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.341617    4528 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-xbkcj_cac9a7a9-12dc-4699-bf59-da9a169f261d/konnectivity-agent/0.log" limit=1024
Nov 03 05:55:13.461972 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.461934    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:13.462138 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.462014    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:13.462138 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.462050    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:13.462138 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462100    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:13.462262 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462152    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:13.462262 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462168    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:13.462262 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462183    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.46216426 +0000 UTC m=+2.873561320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:13.462262 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462201    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.462192674 +0000 UTC m=+2.873589730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:13.462262 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:13.462216    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:14.462208674 +0000 UTC m=+2.873605730 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:13.785289 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:13.785239    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj" event={"ID":"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b","Type":"ContainerStarted","Data":"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"}
Nov 03 05:55:14.471222 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:14.471181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:14.471222 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:14.471232    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:14.471279    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471349    4528 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471393    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471440    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls podName:2fc905f9-78f6-410f-8736-48ce0bced9d2 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:16.471414985 +0000 UTC m=+4.882812058 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5v8g7" (UID: "2fc905f9-78f6-410f-8736-48ce0bced9d2") : secret "insights-runtime-extractor-tls" not found
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471462    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:16.471449384 +0000 UTC m=+4.882846440 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471478    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:14.471593 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:14.471543    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:16.471525709 +0000 UTC m=+4.882922782 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:15.794526 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:15.794478    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-8gmvq" event={"ID":"a4a33ca8-05a6-48cb-9c4f-b990d1c7ca62","Type":"ContainerStarted","Data":"782186753cb3f844846a1b946f44e91db7daedebaf282016238c8ffdf907b2a6"}
Nov 03 05:55:16.489288 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.489244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:16.489515 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.489319    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:16.489515 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.489351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:16.489515 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:16.489400    4528 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 03 05:55:16.489515 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:16.489458    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:16.489515 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:16.489481    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls podName:7c828422-e7da-4ee5-b748-8e8827dded89 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:20.48945222 +0000 UTC m=+8.900849277 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls") pod "dns-default-p8jsb" (UID: "7c828422-e7da-4ee5-b748-8e8827dded89") : secret "dns-default-metrics-tls" not found
Nov 03 05:55:16.489701 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:16.489531    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:20.489516382 +0000 UTC m=+8.900913453 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:16.489701 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.489681    4528 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 05:55:16.491707 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.491688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/2fc905f9-78f6-410f-8736-48ce0bced9d2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5v8g7\" (UID: \"2fc905f9-78f6-410f-8736-48ce0bced9d2\") " pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:16.629932 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.629892    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5v8g7"
Nov 03 05:55:16.932063 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.932034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5v8g7"]
Nov 03 05:55:16.935407 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:16.935368    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc905f9_78f6_410f_8736_48ce0bced9d2.slice/crio-cfad158071f55a9f76cd8f3cf483b8601cbd5efb41ccfd618f049ca25ad32b60 WatchSource:0}: Error finding container cfad158071f55a9f76cd8f3cf483b8601cbd5efb41ccfd618f049ca25ad32b60: Status 404 returned error can't find the container with id cfad158071f55a9f76cd8f3cf483b8601cbd5efb41ccfd618f049ca25ad32b60
Nov 03 05:55:16.967582 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:16.967558    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xbkcj"
Nov 03 05:55:17.802677 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:17.802646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v8g7" event={"ID":"2fc905f9-78f6-410f-8736-48ce0bced9d2","Type":"ContainerStarted","Data":"2cf365a21ba66a540366b3d7ed6bee2054bf0192b28b6a4b68c5821d00b61a88"}
Nov 03 05:55:17.802837 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:17.802701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v8g7" event={"ID":"2fc905f9-78f6-410f-8736-48ce0bced9d2","Type":"ContainerStarted","Data":"df3b48167588ec426f2ac4ea4cd9133d5287fb02ced9896e21e5a6b0849b942a"}
Nov 03 05:55:17.802837 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:17.802720    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v8g7" event={"ID":"2fc905f9-78f6-410f-8736-48ce0bced9d2","Type":"ContainerStarted","Data":"cfad158071f55a9f76cd8f3cf483b8601cbd5efb41ccfd618f049ca25ad32b60"}
Nov 03 05:55:19.424388 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:19.424362    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvprb_81d514bd-df1e-428c-ab18-66f841c88928/dns-node-resolver/0.log"
Nov 03 05:55:19.811672 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:19.811634    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5v8g7" event={"ID":"2fc905f9-78f6-410f-8736-48ce0bced9d2","Type":"ContainerStarted","Data":"b0e0e4c90de5a1175205a9da953566f6edd564d9072142d2301a3c5fb7eb9740"}
Nov 03 05:55:20.025432 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.025401    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bws8t_23536828-f0dc-4f42-8f23-988d8de6a262/node-ca/0.log"
Nov 03 05:55:20.509327 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.509301    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:20.516212 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.516180    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:20.516361 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.516265    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:20.516402 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:20.516359    4528 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 03 05:55:20.516435 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:20.516423    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs podName:87a29b40-9e0d-4dde-bedb-c9423919e268 nodeName:}" failed. No retries permitted until 2026-11-03 05:55:28.516406308 +0000 UTC m=+16.927803367 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs") pod "network-metrics-daemon-glzh2" (UID: "87a29b40-9e0d-4dde-bedb-c9423919e268") : secret "metrics-daemon-secret" not found
Nov 03 05:55:20.518403 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.518385    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7c828422-e7da-4ee5-b748-8e8827dded89-metrics-tls\") pod \"dns-default-p8jsb\" (UID: \"7c828422-e7da-4ee5-b748-8e8827dded89\") " pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:20.527903 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.527881    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:20.819897 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:20.819863    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p8jsb"]
Nov 03 05:55:20.824636 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:20.824610    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c828422_e7da_4ee5_b748_8e8827dded89.slice/crio-5692c84ba630fdc18838335f4505a9d9136f94377105f728f1d90ee81162650c WatchSource:0}: Error finding container 5692c84ba630fdc18838335f4505a9d9136f94377105f728f1d90ee81162650c: Status 404 returned error can't find the container with id 5692c84ba630fdc18838335f4505a9d9136f94377105f728f1d90ee81162650c
Nov 03 05:55:21.224087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:21.224020    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hxvcn_87999404-7a09-46b3-8d41-65407bb93863/serve-healthcheck-canary/0.log"
Nov 03 05:55:21.625524 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:21.625493    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-hhsb4_19e4e8ea-dd59-4bcb-924c-870fd4392bcf/migrator/0.log"
Nov 03 05:55:21.818103 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:21.818061    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8jsb" event={"ID":"7c828422-e7da-4ee5-b748-8e8827dded89","Type":"ContainerStarted","Data":"5692c84ba630fdc18838335f4505a9d9136f94377105f728f1d90ee81162650c"}
Nov 03 05:55:22.220863 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:22.220833    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:22.221466 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:22.221425    4528 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 05:55:22.822934 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:22.822899    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8jsb" event={"ID":"7c828422-e7da-4ee5-b748-8e8827dded89","Type":"ContainerStarted","Data":"ac0a9e0826d14c1c1a2b5e672fecb8d4a111e0a5b18013751925c7b52e5eaa67"}
Nov 03 05:55:22.823411 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:22.822987    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p8jsb" event={"ID":"7c828422-e7da-4ee5-b748-8e8827dded89","Type":"ContainerStarted","Data":"61ae5c82cb8d3020ad196f99456c1c2ba84bd396a19bc701e41184635d8b57a0"}
Nov 03 05:55:22.823411 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:22.823046    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:28.579539 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:28.579496    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:28.581762 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:28.581738    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/87a29b40-9e0d-4dde-bedb-c9423919e268-metrics-certs\") pod \"network-metrics-daemon-glzh2\" (UID: \"87a29b40-9e0d-4dde-bedb-c9423919e268\") " pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:28.624833 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:28.624807    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-glzh2"
Nov 03 05:55:28.912721 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:28.912650    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-glzh2"]
Nov 03 05:55:28.915941 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:28.915917    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a29b40_9e0d_4dde_bedb_c9423919e268.slice/crio-8856d93dc3b3a34bfa0ef1d69f9c2685a842fff6f70385466d050283e21c01f0 WatchSource:0}: Error finding container 8856d93dc3b3a34bfa0ef1d69f9c2685a842fff6f70385466d050283e21c01f0: Status 404 returned error can't find the container with id 8856d93dc3b3a34bfa0ef1d69f9c2685a842fff6f70385466d050283e21c01f0
Nov 03 05:55:29.844897 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:29.844870    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glzh2" event={"ID":"87a29b40-9e0d-4dde-bedb-c9423919e268","Type":"ContainerStarted","Data":"8ddb6038f6027c2cfd0feb762edfdbd6592bb5cc4fd9a9bf042aefdaa0defb96"}
Nov 03 05:55:29.845211 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:29.844920    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glzh2" event={"ID":"87a29b40-9e0d-4dde-bedb-c9423919e268","Type":"ContainerStarted","Data":"8856d93dc3b3a34bfa0ef1d69f9c2685a842fff6f70385466d050283e21c01f0"}
Nov 03 05:55:30.669663 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.669631    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7b94465567-lmkz8"]
Nov 03 05:55:30.671567 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.671546    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.710683 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.710649    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 03 05:55:30.721411 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.721380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b94465567-lmkz8"]
Nov 03 05:55:30.782781 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.782750    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 03 05:55:30.783095 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.783070    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 03 05:55:30.783224 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.783115    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-97pnm\""
Nov 03 05:55:30.783224 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.783124    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 03 05:55:30.796149 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796124    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-image-registry-private-configuration\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796251 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796164    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-installation-pull-secrets\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796251 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796186    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-bound-sa-token\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796334 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796245    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-tls\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796371 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796350    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-certificates\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796379    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-trusted-ca\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796402 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxmx2\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-kube-api-access-lxmx2\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.796542 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.796523    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-ca-trust-extracted\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.834416 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.834391    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hthhm"
Nov 03 05:55:30.851172 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.851140    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-glzh2" event={"ID":"87a29b40-9e0d-4dde-bedb-c9423919e268","Type":"ContainerStarted","Data":"1ffe5871bf93a4d10dcafcbd3bee1677415865cad192b27cb799d9613b19065b"}
Nov 03 05:55:30.897842 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.897814    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-ca-trust-extracted\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.897984 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.897856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-image-registry-private-configuration\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898064 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.897993    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-installation-pull-secrets\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898064 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898031    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-bound-sa-token\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-tls\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-certificates\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898114    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-trusted-ca\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898304 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898253    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxmx2\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-kube-api-access-lxmx2\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898419 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898385    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-ca-trust-extracted\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.898919 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.898898    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-certificates\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.899267 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.899247    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-trusted-ca\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.900475 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.900450    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-installation-pull-secrets\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.900834 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.900814    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-image-registry-private-configuration\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.900958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.900939    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-registry-tls\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.920544 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.920476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-bound-sa-token\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.920544 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.920504    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxmx2\" (UniqueName: \"kubernetes.io/projected/9f0c653e-c147-4c5d-8c46-9e9db71a9feb-kube-api-access-lxmx2\") pod \"image-registry-7b94465567-lmkz8\" (UID: \"9f0c653e-c147-4c5d-8c46-9e9db71a9feb\") " pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:30.983998 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:30.983968    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:31.290482 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:31.290415    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7b94465567-lmkz8"]
Nov 03 05:55:31.293278 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:31.293252    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f0c653e_c147_4c5d_8c46_9e9db71a9feb.slice/crio-797c9bd88c9490f0247a88290c1f6b78ecf9c9a34467c55172826163af6c08f8 WatchSource:0}: Error finding container 797c9bd88c9490f0247a88290c1f6b78ecf9c9a34467c55172826163af6c08f8: Status 404 returned error can't find the container with id 797c9bd88c9490f0247a88290c1f6b78ecf9c9a34467c55172826163af6c08f8
Nov 03 05:55:31.855766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:31.855731    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94465567-lmkz8" event={"ID":"9f0c653e-c147-4c5d-8c46-9e9db71a9feb","Type":"ContainerStarted","Data":"1e1d2c6793653e7b80472b9e1f615f046a090936be7af5d64ef8beebc882f301"}
Nov 03 05:55:31.855766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:31.855780    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7b94465567-lmkz8" event={"ID":"9f0c653e-c147-4c5d-8c46-9e9db71a9feb","Type":"ContainerStarted","Data":"797c9bd88c9490f0247a88290c1f6b78ecf9c9a34467c55172826163af6c08f8"}
Nov 03 05:55:31.901145 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:31.901080    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7b94465567-lmkz8" podStartSLOduration=1.90106199 podStartE2EDuration="1.90106199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:31.899623967 +0000 UTC m=+20.311021044" watchObservedRunningTime="2026-11-03 05:55:31.90106199 +0000 UTC m=+20.312459069"
Nov 03 05:55:32.829210 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:32.829178    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p8jsb"
Nov 03 05:55:32.859168 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:32.859131    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:38.286855 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.286817    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"]
Nov 03 05:55:38.290473 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.290451    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.316858 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.316828    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 05:55:38.316858 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.316848    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 05:55:38.316858 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.316868    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 05:55:38.317073 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.316828    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-hvvfj\""
Nov 03 05:55:38.317073 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.317066    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 05:55:38.317155 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.317098    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 05:55:38.318162 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.318140    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"]
Nov 03 05:55:38.333017 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.332993    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"]
Nov 03 05:55:38.335167 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.335153    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.353344 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.353318    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k96cf"]
Nov 03 05:55:38.355927 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.355907    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.359126 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.359100    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"]
Nov 03 05:55:38.387436 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.387404    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 05:55:38.387633 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.387614    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 05:55:38.388136 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.388110    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-s5bbp\""
Nov 03 05:55:38.395837 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.395821    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 05:55:38.395908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.395851    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 05:55:38.395965 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.395911    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-cbdsf\""
Nov 03 05:55:38.395965 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.395938    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 05:55:38.396031 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.395969    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 05:55:38.444326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444276    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxt8f\" (UniqueName: \"kubernetes.io/projected/8f65d210-43a3-41de-a608-1bf820be5620-kube-api-access-pxt8f\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444326 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444327    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e398089-1045-4b35-bcad-f9649b94a78c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.444508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444402    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.444508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444444    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-accelerators-collector-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444501    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c090e4d-9e88-49f2-96fc-6f3876ab1928-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.444606 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444551    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.444606 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444580    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-textfile\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444606 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444594    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhbnn\" (UniqueName: \"kubernetes.io/projected/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-api-access-rhbnn\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.444690 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444615    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-metrics-client-ca\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444690 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444654    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.444751 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444689    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.444823 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444807    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-tls\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444866 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444834    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-sys\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444866 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444850    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-root\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.444929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444892    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z625\" (UniqueName: \"kubernetes.io/projected/1e398089-1045-4b35-bcad-f9649b94a78c-kube-api-access-7z625\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.444929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.445018 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444951    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-wtmp\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.445018 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.444968    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.545729 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545642    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.545729 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545682    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-textfile\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.545729 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545708    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhbnn\" (UniqueName: \"kubernetes.io/projected/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-api-access-rhbnn\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545736    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-metrics-client-ca\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545758    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545784    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545865    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-tls\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545891    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-sys\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545916    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-root\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545939    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545963    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z625\" (UniqueName: \"kubernetes.io/projected/1e398089-1045-4b35-bcad-f9649b94a78c-kube-api-access-7z625\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.545992    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546017    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-wtmp\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546041    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546088    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxt8f\" (UniqueName: \"kubernetes.io/projected/8f65d210-43a3-41de-a608-1bf820be5620-kube-api-access-pxt8f\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546117    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e398089-1045-4b35-bcad-f9649b94a78c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546126    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-textfile\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546151    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546196    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-accelerators-collector-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546222    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c090e4d-9e88-49f2-96fc-6f3876ab1928-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546490 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546271    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-wtmp\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.546908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.546908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.546521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/1c090e4d-9e88-49f2-96fc-6f3876ab1928-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.547112 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-metrics-client-ca\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.547223 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547195    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.547419 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-sys\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.547535 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547506    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-accelerators-collector-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.547612 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547592    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8f65d210-43a3-41de-a608-1bf820be5620-root\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.547785 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.547757    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1e398089-1045-4b35-bcad-f9649b94a78c-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.548812 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.548739    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.548812 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.548760    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.548955 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.548824    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1e398089-1045-4b35-bcad-f9649b94a78c-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.549166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.549144    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.549293 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.549277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.549956 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.549941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8f65d210-43a3-41de-a608-1bf820be5620-node-exporter-tls\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.568341 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.568318    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxt8f\" (UniqueName: \"kubernetes.io/projected/8f65d210-43a3-41de-a608-1bf820be5620-kube-api-access-pxt8f\") pod \"node-exporter-k96cf\" (UID: \"8f65d210-43a3-41de-a608-1bf820be5620\") " pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.568441 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.568410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhbnn\" (UniqueName: \"kubernetes.io/projected/1c090e4d-9e88-49f2-96fc-6f3876ab1928-kube-api-access-rhbnn\") pod \"kube-state-metrics-5dc8d7d89b-qmc2c\" (UID: \"1c090e4d-9e88-49f2-96fc-6f3876ab1928\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.568656 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.568634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z625\" (UniqueName: \"kubernetes.io/projected/1e398089-1045-4b35-bcad-f9649b94a78c-kube-api-access-7z625\") pod \"openshift-state-metrics-7bd7489957-c2c5q\" (UID: \"1e398089-1045-4b35-bcad-f9649b94a78c\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.603325 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.603291    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"
Nov 03 05:55:38.648939 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.648909    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"
Nov 03 05:55:38.671471 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.671213    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k96cf"
Nov 03 05:55:38.681587 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:38.681556    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f65d210_43a3_41de_a608_1bf820be5620.slice/crio-81008a62d2063677240a043022094cb307d9070d163c7a07de04cc3737ce3bc1 WatchSource:0}: Error finding container 81008a62d2063677240a043022094cb307d9070d163c7a07de04cc3737ce3bc1: Status 404 returned error can't find the container with id 81008a62d2063677240a043022094cb307d9070d163c7a07de04cc3737ce3bc1
Nov 03 05:55:38.877672 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.877639    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k96cf" event={"ID":"8f65d210-43a3-41de-a608-1bf820be5620","Type":"ContainerStarted","Data":"81008a62d2063677240a043022094cb307d9070d163c7a07de04cc3737ce3bc1"}
Nov 03 05:55:38.907456 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.907418    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q"]
Nov 03 05:55:38.910189 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:38.910160    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e398089_1045_4b35_bcad_f9649b94a78c.slice/crio-4cd635c0c288f76865b8a340a5b856af35cb8b21fc9e31690712d025280af71b WatchSource:0}: Error finding container 4cd635c0c288f76865b8a340a5b856af35cb8b21fc9e31690712d025280af71b: Status 404 returned error can't find the container with id 4cd635c0c288f76865b8a340a5b856af35cb8b21fc9e31690712d025280af71b
Nov 03 05:55:38.948883 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:38.948855    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c"]
Nov 03 05:55:38.951943 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:38.951906    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c090e4d_9e88_49f2_96fc_6f3876ab1928.slice/crio-73d0822e0f284e0e9201e618728cf0dd0c411a42f64ea4fbfe0c7e24ff97c72b WatchSource:0}: Error finding container 73d0822e0f284e0e9201e618728cf0dd0c411a42f64ea4fbfe0c7e24ff97c72b: Status 404 returned error can't find the container with id 73d0822e0f284e0e9201e618728cf0dd0c411a42f64ea4fbfe0c7e24ff97c72b
Nov 03 05:55:39.883445 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.883367    4528 generic.go:356] "Generic (PLEG): container finished" podID="8f65d210-43a3-41de-a608-1bf820be5620" containerID="36f5ae5d56bc626999aeff1d72ed96ab4a9ade7aa5b4c40236693e97bda44121" exitCode=0
Nov 03 05:55:39.883924 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.883456    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k96cf" event={"ID":"8f65d210-43a3-41de-a608-1bf820be5620","Type":"ContainerDied","Data":"36f5ae5d56bc626999aeff1d72ed96ab4a9ade7aa5b4c40236693e97bda44121"}
Nov 03 05:55:39.886315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.886285    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q" event={"ID":"1e398089-1045-4b35-bcad-f9649b94a78c","Type":"ContainerStarted","Data":"b4734c1e510164ff0e1ffae36f3724c1a8a2455671cfa3af29c9216399993485"}
Nov 03 05:55:39.886434 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.886346    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q" event={"ID":"1e398089-1045-4b35-bcad-f9649b94a78c","Type":"ContainerStarted","Data":"75d1a3c53db7ad5b455e98e565eee862fa8d475455c430a6f0493c95a095d05f"}
Nov 03 05:55:39.886434 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.886372    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q" event={"ID":"1e398089-1045-4b35-bcad-f9649b94a78c","Type":"ContainerStarted","Data":"4cd635c0c288f76865b8a340a5b856af35cb8b21fc9e31690712d025280af71b"}
Nov 03 05:55:39.887666 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:39.887641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c" event={"ID":"1c090e4d-9e88-49f2-96fc-6f3876ab1928","Type":"ContainerStarted","Data":"73d0822e0f284e0e9201e618728cf0dd0c411a42f64ea4fbfe0c7e24ff97c72b"}
Nov 03 05:55:40.893491 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.893451    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c" event={"ID":"1c090e4d-9e88-49f2-96fc-6f3876ab1928","Type":"ContainerStarted","Data":"8d43b199acde28a1fa3943c69078760a162d6d76afe051247731326d3fd48576"}
Nov 03 05:55:40.893990 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.893528    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c" event={"ID":"1c090e4d-9e88-49f2-96fc-6f3876ab1928","Type":"ContainerStarted","Data":"7050aa1ff7ddb286a328a144c9a0c4950d674ca8fe5e6139837403f2360877c4"}
Nov 03 05:55:40.893990 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.893553    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c" event={"ID":"1c090e4d-9e88-49f2-96fc-6f3876ab1928","Type":"ContainerStarted","Data":"a8bad24368b0ed38b2d5efb51b56c6f444840c922ec69fe637a812e0353e695a"}
Nov 03 05:55:40.895695 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.895663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k96cf" event={"ID":"8f65d210-43a3-41de-a608-1bf820be5620","Type":"ContainerStarted","Data":"4722889ab2aeeb77d373b3dadfc67505744cad862f26d748b2ad3c061f423d29"}
Nov 03 05:55:40.895833 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.895714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k96cf" event={"ID":"8f65d210-43a3-41de-a608-1bf820be5620","Type":"ContainerStarted","Data":"e3507084a508cf0475957d1dbab92c2cfc928f38b507e18ea85615e59bea0485"}
Nov 03 05:55:40.897696 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.897670    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q" event={"ID":"1e398089-1045-4b35-bcad-f9649b94a78c","Type":"ContainerStarted","Data":"491735e89dafb3a3d05392576d553ddffdbfd869918535dfa569ecb7905922f0"}
Nov 03 05:55:40.935860 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:40.935809    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-qmc2c" podStartSLOduration=1.723500437 podStartE2EDuration="2.935779177s" totalImagesPullingTime="1.21227874s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:40.934953505 +0000 UTC m=+29.346350594" watchObservedRunningTime="2026-11-03 05:55:40.935779177 +0000 UTC m=+29.347176252"
Nov 03 05:55:41.005002 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:41.004939    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-c2c5q" podStartSLOduration=2.088912341 podStartE2EDuration="3.004921295s" totalImagesPullingTime="916.008954ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:41.004050246 +0000 UTC m=+29.415447322" watchObservedRunningTime="2026-11-03 05:55:41.004921295 +0000 UTC m=+29.416318373"
Nov 03 05:55:41.052409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:41.052351    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k96cf" podStartSLOduration=2.067739982 podStartE2EDuration="3.052335357s" totalImagesPullingTime="905.409957ms" totalInitContainerRuntime="79.185418ms" isStatefulPod=true podCreationTimestamp="2026-11-03 05:55:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:41.050622255 +0000 UTC m=+29.462019341" watchObservedRunningTime="2026-11-03 05:55:41.052335357 +0000 UTC m=+29.463732678"
Nov 03 05:55:43.010211 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.010174    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:55:43.041746 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.041716    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:55:43.041924 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.041858    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.126427 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126396    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 05:55:43.126888 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126841    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 05:55:43.127043 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126894    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 05:55:43.127043 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126896    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tw5zl\""
Nov 03 05:55:43.127043 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126917    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 05:55:43.127043 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.126913    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 05:55:43.127218 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.127122    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 05:55:43.150635 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.150610    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 05:55:43.150826 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.150808    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 05:55:43.180275 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180237    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180275 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180278    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk4q5\" (UniqueName: \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180519 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180323    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180519 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180519 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180428    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180609 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180532    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180609 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180581    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.180704 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.180615    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.281673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281590    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.281673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.281673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281655    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.281673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk4q5\" (UniqueName: \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282028 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281701    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282028 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281871    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282028 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281918    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282028 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.281949    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282386 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.282362    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282465 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.282449    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.282625    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.282632    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.282687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.282630    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.284180 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.284159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.284357 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.284338    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.304489 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.304457    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk4q5\" (UniqueName: \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\") pod \"console-5df85fd9f7-d6p8x\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") " pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.366316 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.366272    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:43.614936 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.614904    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"]
Nov 03 05:55:43.650749 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.650715    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"]
Nov 03 05:55:43.650917 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.650876    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.666216 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666159    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 05:55:43.666216 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666169    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 05:55:43.666427 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666218    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 05:55:43.666427 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666273    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 05:55:43.666427 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666408    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 05:55:43.666572 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.666477    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fmsgq\""
Nov 03 05:55:43.678483 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.678459    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 05:55:43.691435 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:55:43.691401    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:55:43.692307 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.692269    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:55:43.696452 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:43.696429    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd28aa10f_036a_4b6d_820a_59a883bbd1d2.slice/crio-54d26a5abd53ed9fc5e6f4df01b27a4e8c5d60f5170187918ecab67dd90ad6f6 WatchSource:0}: Error finding container 54d26a5abd53ed9fc5e6f4df01b27a4e8c5d60f5170187918ecab67dd90ad6f6: Status 404 returned error can't find the container with id 54d26a5abd53ed9fc5e6f4df01b27a4e8c5d60f5170187918ecab67dd90ad6f6
Nov 03 05:55:43.785006 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.784969    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-federate-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785010    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785108    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rgn\" (UniqueName: \"kubernetes.io/projected/63835035-86c3-4afd-ac62-7eaa1165e159-kube-api-access-x6rgn\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785166 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785144    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-metrics-client-ca\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785166    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785239    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.785446 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.785345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.787577 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.787555    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-g5slj"
Nov 03 05:55:43.886303 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886229    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886429 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886276    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886429 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886330    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-federate-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886429 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886350    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886569 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886541    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rgn\" (UniqueName: \"kubernetes.io/projected/63835035-86c3-4afd-ac62-7eaa1165e159-kube-api-access-x6rgn\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886627 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886592    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-metrics-client-ca\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886675 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.886675 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.886658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.887254 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.887190    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-serving-certs-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.887455 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.887423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-metrics-client-ca\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.887561 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.887527    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.888917 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.888891    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-federate-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.889008 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.888980    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-telemeter-client-tls\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.889101 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.889084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.889279 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.889259    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/63835035-86c3-4afd-ac62-7eaa1165e159-secret-telemeter-client\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:43.909196 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.909162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df85fd9f7-d6p8x" event={"ID":"d28aa10f-036a-4b6d-820a-59a883bbd1d2","Type":"ContainerStarted","Data":"54d26a5abd53ed9fc5e6f4df01b27a4e8c5d60f5170187918ecab67dd90ad6f6"}
Nov 03 05:55:43.909296 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:43.909248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rgn\" (UniqueName: \"kubernetes.io/projected/63835035-86c3-4afd-ac62-7eaa1165e159-kube-api-access-x6rgn\") pod \"telemeter-client-6f7bd556c7-rpgfk\" (UID: \"63835035-86c3-4afd-ac62-7eaa1165e159\") " pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:44.000089 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:44.000060    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"
Nov 03 05:55:44.345956 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:44.345811    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk"]
Nov 03 05:55:44.349173 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:55:44.349143    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63835035_86c3_4afd_ac62_7eaa1165e159.slice/crio-51087b596b17dc364aa6e48c509cedc8983b2e69a00cf8524f1de09e303fda7d WatchSource:0}: Error finding container 51087b596b17dc364aa6e48c509cedc8983b2e69a00cf8524f1de09e303fda7d: Status 404 returned error can't find the container with id 51087b596b17dc364aa6e48c509cedc8983b2e69a00cf8524f1de09e303fda7d
Nov 03 05:55:44.917677 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:44.917639    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk" event={"ID":"63835035-86c3-4afd-ac62-7eaa1165e159","Type":"ContainerStarted","Data":"51087b596b17dc364aa6e48c509cedc8983b2e69a00cf8524f1de09e303fda7d"}
Nov 03 05:55:46.925672 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:46.925644    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk" event={"ID":"63835035-86c3-4afd-ac62-7eaa1165e159","Type":"ContainerStarted","Data":"b10651f35918d80a5773e77f9fbeff9b7ef4302e5c8cda116d3957c3d55d868f"}
Nov 03 05:55:47.931602 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:47.931513    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk" event={"ID":"63835035-86c3-4afd-ac62-7eaa1165e159","Type":"ContainerStarted","Data":"1dd7403d90c61e049ebd21076dea971be09db6ab0ecf22a6c2b42e1b28511eba"}
Nov 03 05:55:47.931602 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:47.931591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk" event={"ID":"63835035-86c3-4afd-ac62-7eaa1165e159","Type":"ContainerStarted","Data":"b1bdf30dcc22f7dba56718093e52158e439ebec0fe687bc31bf41aaf78b33ad6"}
Nov 03 05:55:47.933129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:47.933106    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df85fd9f7-d6p8x" event={"ID":"d28aa10f-036a-4b6d-820a-59a883bbd1d2","Type":"ContainerStarted","Data":"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"}
Nov 03 05:55:48.001517 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:48.001467    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6f7bd556c7-rpgfk" podStartSLOduration=1.814895778 podStartE2EDuration="5.001453053s" totalImagesPullingTime="3.186557275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:47.989390795 +0000 UTC m=+36.400787874" watchObservedRunningTime="2026-11-03 05:55:48.001453053 +0000 UTC m=+36.412850316"
Nov 03 05:55:48.036715 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:48.036654    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5df85fd9f7-d6p8x" podStartSLOduration=2.814736624 podStartE2EDuration="6.036635315s" totalImagesPullingTime="3.221898691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:55:48.036241689 +0000 UTC m=+36.447638766" watchObservedRunningTime="2026-11-03 05:55:48.036635315 +0000 UTC m=+36.448032396"
Nov 03 05:55:53.366641 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.366604    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:53.367146 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.366714    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:53.371357 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.371334    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:55:53.426050 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.426013    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:55:53.866391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.866360    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7b94465567-lmkz8"
Nov 03 05:55:53.959102 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:55:53.959075    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:56:13.690233 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:56:13.690191    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:56:19.978854 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:19.978776    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5df85fd9f7-d6p8x" podUID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" containerName="console" containerID="cri-o://a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48" gracePeriod=15
Nov 03 05:56:20.271942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.271912    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:56:20.381514 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.381469    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.381813 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.381771    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.381996 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.381961    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.382134 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.382119    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\" (UniqueName: \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.382236 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.382224    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.382329 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.382317    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.382435 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.382422    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.382545 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.382533    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\") pod \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\" (UID: \"d28aa10f-036a-4b6d-820a-59a883bbd1d2\") "
Nov 03 05:56:20.383270 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.383244    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:20.383752 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.383732    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:20.384077 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.384061    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:56:20.387116 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.386836    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 05:56:20.389128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.389091    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:20.389128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.389103    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:20.389898 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.389377    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "kube-api-access-fk4q5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:56:20.390720 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.390695    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert" pod "d28aa10f-036a-4b6d-820a-59a883bbd1d2" (UID: "d28aa10f-036a-4b6d-820a-59a883bbd1d2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 05:56:20.483461 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483430    4528 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-serving-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483461 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483455    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-trusted-ca-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483461 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483465    4528 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-oauth-serving-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483475    4528 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d28aa10f-036a-4b6d-820a-59a883bbd1d2-tmp\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483485    4528 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-oauth-config\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483496    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk4q5\" (UniqueName: \"kubernetes.io/projected/d28aa10f-036a-4b6d-820a-59a883bbd1d2-kube-api-access-fk4q5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483506    4528 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-console-config\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:20.483687 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:20.483513    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d28aa10f-036a-4b6d-820a-59a883bbd1d2-service-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:21.045285 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.045251    4528 generic.go:356] "Generic (PLEG): container finished" podID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" containerID="a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48" exitCode=0
Nov 03 05:56:21.045698 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.045324    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5df85fd9f7-d6p8x"
Nov 03 05:56:21.045698 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.045338    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df85fd9f7-d6p8x" event={"ID":"d28aa10f-036a-4b6d-820a-59a883bbd1d2","Type":"ContainerDied","Data":"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"}
Nov 03 05:56:21.045698 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.045399    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5df85fd9f7-d6p8x" event={"ID":"d28aa10f-036a-4b6d-820a-59a883bbd1d2","Type":"ContainerDied","Data":"54d26a5abd53ed9fc5e6f4df01b27a4e8c5d60f5170187918ecab67dd90ad6f6"}
Nov 03 05:56:21.045698 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.045430    4528 scope.go:122] "RemoveContainer" containerID="a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"
Nov 03 05:56:21.053935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.053914    4528 scope.go:122] "RemoveContainer" containerID="a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"
Nov 03 05:56:21.054230 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:56:21.054211    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48\": container with ID starting with a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48 not found: ID does not exist" containerID="a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"
Nov 03 05:56:21.054285 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.054239    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48"} err="failed to get container status \"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48\": rpc error: code = NotFound desc = could not find container \"a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48\": container with ID starting with a3bbf7ccc5921c20d36e11963e73cc0d6468f48ce67c3b87441a274a05d62c48 not found: ID does not exist"
Nov 03 05:56:21.094627 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.094592    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:56:21.106068 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.106039    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5df85fd9f7-d6p8x"]
Nov 03 05:56:21.745660 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:21.745627    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" path="/var/lib/kubelet/pods/d28aa10f-036a-4b6d-820a-59a883bbd1d2/volumes"
Nov 03 05:56:25.228004 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.227970    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-74b667d744-r9f6w"]
Nov 03 05:56:25.228489 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.228216    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" containerName="console"
Nov 03 05:56:25.228489 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.228226    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" containerName="console"
Nov 03 05:56:25.228489 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.228276    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d28aa10f-036a-4b6d-820a-59a883bbd1d2" containerName="console"
Nov 03 05:56:25.230236 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.230217    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.284935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.284909    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 05:56:25.284935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.284909    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 05:56:25.284935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.284909    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 05:56:25.286165 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.286144    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b667d744-r9f6w"]
Nov 03 05:56:25.295515 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.295494    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 05:56:25.295626 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.295559    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 05:56:25.296455 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.296434    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 05:56:25.296514 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.296494    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tw5zl\""
Nov 03 05:56:25.296514 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.296494    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 05:56:25.320878 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.320852    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 05:56:25.323725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323705    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-trusted-ca-bundle\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.323788 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6php7\" (UniqueName: \"kubernetes.io/projected/2a33c2da-9397-440c-aaef-78808c182b6c-kube-api-access-6php7\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.323788 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323758    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-service-ca\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.323923 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323905    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-console-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.323966 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323931    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-oauth-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.324011 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.323997    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.324048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.324029    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a33c2da-9397-440c-aaef-78808c182b6c-tmp\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.324089 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.324073    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-oauth-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425393    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a33c2da-9397-440c-aaef-78808c182b6c-tmp\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425429    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-oauth-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425464    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-trusted-ca-bundle\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425525    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6php7\" (UniqueName: \"kubernetes.io/projected/2a33c2da-9397-440c-aaef-78808c182b6c-kube-api-access-6php7\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425558    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-service-ca\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425641    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-console-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.425671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-oauth-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.426827 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.426577    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-oauth-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.430126 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.430095    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a33c2da-9397-440c-aaef-78808c182b6c-tmp\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.430395 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.430369    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-service-ca\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.430756 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.430734    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-serving-cert\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.430847 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.430764    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-console-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.430908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.430886    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a33c2da-9397-440c-aaef-78808c182b6c-trusted-ca-bundle\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.431534 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.431502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a33c2da-9397-440c-aaef-78808c182b6c-console-oauth-config\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.452419 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.452374    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6php7\" (UniqueName: \"kubernetes.io/projected/2a33c2da-9397-440c-aaef-78808c182b6c-kube-api-access-6php7\") pod \"console-74b667d744-r9f6w\" (UID: \"2a33c2da-9397-440c-aaef-78808c182b6c\") " pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.540717 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.540623    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:25.834435 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:25.834405    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-74b667d744-r9f6w"]
Nov 03 05:56:25.837525 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:56:25.837500    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a33c2da_9397_440c_aaef_78808c182b6c.slice/crio-048b2d6e3bc78825841a4dd57b252f6b45dba0eaeb5328eb72d831ed75a4b811 WatchSource:0}: Error finding container 048b2d6e3bc78825841a4dd57b252f6b45dba0eaeb5328eb72d831ed75a4b811: Status 404 returned error can't find the container with id 048b2d6e3bc78825841a4dd57b252f6b45dba0eaeb5328eb72d831ed75a4b811
Nov 03 05:56:26.066670 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:26.066636    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b667d744-r9f6w" event={"ID":"2a33c2da-9397-440c-aaef-78808c182b6c","Type":"ContainerStarted","Data":"b02030735a4ec5b7adea510b35828e14e4d10d529b16311699c4773e1b7ae357"}
Nov 03 05:56:26.066857 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:26.066685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-74b667d744-r9f6w" event={"ID":"2a33c2da-9397-440c-aaef-78808c182b6c","Type":"ContainerStarted","Data":"048b2d6e3bc78825841a4dd57b252f6b45dba0eaeb5328eb72d831ed75a4b811"}
Nov 03 05:56:26.109038 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:26.108936    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-74b667d744-r9f6w" podStartSLOduration=1.108922379 podStartE2EDuration="1.108922379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:56:26.108057994 +0000 UTC m=+74.519455072" watchObservedRunningTime="2026-11-03 05:56:26.108922379 +0000 UTC m=+74.520319457"
Nov 03 05:56:31.282109 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.282074    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 05:56:31.285018 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.284997    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.300054 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.300026    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 05:56:31.301185 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.301163    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 05:56:31.311170 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311142    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 05:56:31.311323 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311200    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 05:56:31.311323 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311216    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 05:56:31.311323 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311217    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 05:56:31.311323 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311199    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 05:56:31.311546 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311532    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 05:56:31.311654 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311637    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 05:56:31.311710 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.311692    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-mt6jq\""
Nov 03 05:56:31.323704 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.323680    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 05:56:31.369930 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.369893    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370097 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.369939    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370097 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.369963    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370097 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370032    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-web-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370191 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370191 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370135    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370252 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-config-volume\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370252 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370219    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc8bh\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-kube-api-access-fc8bh\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370252 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370237    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-config-out\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370338 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370289    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370338 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370307    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370401 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370356    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.370401 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.370377    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471196 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471314 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471223    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471314 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471314 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471294    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471319    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-web-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471344    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471391    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-config-volume\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471413    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc8bh\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-kube-api-access-fc8bh\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471648 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471439    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-config-out\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471648 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471475    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471648 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471498    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.471648 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471533    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.472016 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.471996    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.472937 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.472910    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474086 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/17cc8e85-12a6-4449-9765-bed369222867-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474466 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474434    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474565 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474462    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/17cc8e85-12a6-4449-9765-bed369222867-config-out\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474565 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474642 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474628    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474819 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474769    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-tls-assets\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.474873 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.474803    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.475315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.475292    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-web-config\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.476479 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.476450    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.476917 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.476894    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/17cc8e85-12a6-4449-9765-bed369222867-config-volume\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.498094 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.498061    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc8bh\" (UniqueName: \"kubernetes.io/projected/17cc8e85-12a6-4449-9765-bed369222867-kube-api-access-fc8bh\") pod \"alertmanager-main-0\" (UID: \"17cc8e85-12a6-4449-9765-bed369222867\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.603929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.603892    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 05:56:31.931895 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:31.931812    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 05:56:31.934857 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:56:31.934823    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17cc8e85_12a6_4449_9765_bed369222867.slice/crio-b151181323045c97267c4ab94edcabb6e24571988992064bdba1cfa01e89782d WatchSource:0}: Error finding container b151181323045c97267c4ab94edcabb6e24571988992064bdba1cfa01e89782d: Status 404 returned error can't find the container with id b151181323045c97267c4ab94edcabb6e24571988992064bdba1cfa01e89782d
Nov 03 05:56:32.088120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:32.088087    4528 generic.go:356] "Generic (PLEG): container finished" podID="17cc8e85-12a6-4449-9765-bed369222867" containerID="7f9444433b6c36a2a33f438e87e51846bd787789b0836d8c30cf9e69b04d27b5" exitCode=0
Nov 03 05:56:32.088266 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:32.088144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerDied","Data":"7f9444433b6c36a2a33f438e87e51846bd787789b0836d8c30cf9e69b04d27b5"}
Nov 03 05:56:32.088266 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:32.088193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"b151181323045c97267c4ab94edcabb6e24571988992064bdba1cfa01e89782d"}
Nov 03 05:56:34.100925 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.100874    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"4366e02dda01e1621c474e07e7d69ba33969cf28a76dc9f1ebd7c8f8a4dd8bb9"}
Nov 03 05:56:34.101353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.100953    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"e78ae91d503ffce801c3a64f3c58eeaaf4bcab3f35286e33cddf46d6b8e05af2"}
Nov 03 05:56:34.101353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.100993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"d1414430d29261668ae8c8b182a9874c39bbd7f55ebee74508891ed03eaf0aab"}
Nov 03 05:56:34.101353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.101021    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"3e6b2dbec01456dda2976847403bdf838de0a54036d137eaff005f45cbd36fcb"}
Nov 03 05:56:34.101353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.101042    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"24a05636a336032e4ddbfddee33481f3a255cccc03269b9de8f6a16cc53b0506"}
Nov 03 05:56:34.246713 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.246676    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:56:34.246960 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.246919    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj" podUID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" containerName="insights-gathering" containerID="cri-o://766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f" gracePeriod=30
Nov 03 05:56:34.422844 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.422771    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-cg65z-hbq8z"]
Nov 03 05:56:34.425198 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.425173    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.444222 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.444193    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cg65z-hbq8z"]
Nov 03 05:56:34.497389 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.497355    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:56:34.599989 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.599944    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") pod \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") "
Nov 03 05:56:34.600212 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600024    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") pod \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") "
Nov 03 05:56:34.600212 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600052    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") pod \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") "
Nov 03 05:56:34.600212 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600110    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") pod \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\" (UID: \"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b\") "
Nov 03 05:56:34.600212 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600177    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.600464 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600225    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.600464 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.600560 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600503    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path" pod "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" (UID: "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:56:34.600560 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600512    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq9ng\" (UniqueName: \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.600560 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600511    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle" pod "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" (UID: "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:34.600664 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600555    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle" pod "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" (UID: "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:56:34.600664 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600601    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-service-ca-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:34.600664 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.600621    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-archives-path\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:34.602362 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.602341    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5" pod "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" (UID: "f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b"). InnerVolumeSpecName "kube-api-access-6jdd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:56:34.701725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701640    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.701725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701715    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq9ng\" (UniqueName: \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.702001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701742    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.702001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701764    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.702001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701841    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-trusted-ca-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:34.702001 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.701855    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jdd5\" (UniqueName: \"kubernetes.io/projected/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b-kube-api-access-6jdd5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:56:34.702204 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.702068    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.702530 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.702502    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.702606 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.702542    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.724773 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.724745    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq9ng\" (UniqueName: \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\") pod \"periodic-gathering-cg65z-hbq8z\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") " pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:34.738420 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:34.738395    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:56:35.031342 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.031143    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-cg65z-hbq8z"]
Nov 03 05:56:35.033944 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:56:35.033915    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85b7be5_bf2f_431c_89af_c24b14e2712b.slice/crio-a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15 WatchSource:0}: Error finding container a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15: Status 404 returned error can't find the container with id a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15
Nov 03 05:56:35.108625 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.108582    4528 generic.go:356] "Generic (PLEG): container finished" podID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" containerID="766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f" exitCode=2
Nov 03 05:56:35.109121 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.108716    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj" event={"ID":"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b","Type":"ContainerDied","Data":"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"}
Nov 03 05:56:35.109121 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.108968    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj"
Nov 03 05:56:35.109895 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.109827    4528 scope.go:122] "RemoveContainer" containerID="766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"
Nov 03 05:56:35.111379 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.111067    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-z2tsc-b8vnj" event={"ID":"f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b","Type":"ContainerDied","Data":"b33152a5599f9dae1fe76e8c2b316e8bd39cfeff308488900f4ff3f93dad6919"}
Nov 03 05:56:35.115504 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.115462    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"17cc8e85-12a6-4449-9765-bed369222867","Type":"ContainerStarted","Data":"70c54e66fca4a0caf16e76c8884365c3eb4feeef94691662d46181567b4b8838"}
Nov 03 05:56:35.118179 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.118152    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cg65z-hbq8z" event={"ID":"b85b7be5-bf2f-431c-89af-c24b14e2712b","Type":"ContainerStarted","Data":"1472297d00a68be933afb3c94f78e08dad33979085b041d4b3eab5afdec71744"}
Nov 03 05:56:35.118277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.118199    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cg65z-hbq8z" event={"ID":"b85b7be5-bf2f-431c-89af-c24b14e2712b","Type":"ContainerStarted","Data":"a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15"}
Nov 03 05:56:35.122636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.122607    4528 scope.go:122] "RemoveContainer" containerID="766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"
Nov 03 05:56:35.122906 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:56:35.122885    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f\": container with ID starting with 766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f not found: ID does not exist" containerID="766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"
Nov 03 05:56:35.122985 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.122915    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f"} err="failed to get container status \"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f\": rpc error: code = NotFound desc = could not find container \"766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f\": container with ID starting with 766e097ba35edd50464a53a20f944cc7b9800a346c2accd81f02124adb24c15f not found: ID does not exist"
Nov 03 05:56:35.163066 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.163013    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-cg65z-hbq8z" podStartSLOduration=1.163001381 podStartE2EDuration="1.163001381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:56:35.162690116 +0000 UTC m=+83.574087194" watchObservedRunningTime="2026-11-03 05:56:35.163001381 +0000 UTC m=+83.574398456"
Nov 03 05:56:35.230860 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.230809    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.302148288 podStartE2EDuration="4.230778984s" totalImagesPullingTime="1.916961002s" totalInitContainerRuntime="11.669694ms" isStatefulPod=false podCreationTimestamp="2026-11-03 05:56:31 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:56:35.229216141 +0000 UTC m=+83.640613229" watchObservedRunningTime="2026-11-03 05:56:35.230778984 +0000 UTC m=+83.642176060"
Nov 03 05:56:35.265294 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.265228    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:56:35.266747 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.266717    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-z2tsc-b8vnj"]
Nov 03 05:56:35.541220 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.541187    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:35.541410 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.541240    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:35.546365 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.546343    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:35.746514 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:35.746479    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" path="/var/lib/kubelet/pods/f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b/volumes"
Nov 03 05:56:36.126180 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:56:36.126153    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-74b667d744-r9f6w"
Nov 03 05:56:43.693088 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:56:43.693044    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:57:13.690862 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:57:13.690817    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:57:43.691882 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:57:43.691781    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:57:58.421492 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:58.421455    4528 generic.go:356] "Generic (PLEG): container finished" podID="b85b7be5-bf2f-431c-89af-c24b14e2712b" containerID="1472297d00a68be933afb3c94f78e08dad33979085b041d4b3eab5afdec71744" exitCode=0
Nov 03 05:57:58.421492 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:58.421498    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cg65z-hbq8z" event={"ID":"b85b7be5-bf2f-431c-89af-c24b14e2712b","Type":"ContainerDied","Data":"1472297d00a68be933afb3c94f78e08dad33979085b041d4b3eab5afdec71744"}
Nov 03 05:57:59.547483 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.547459    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:57:59.621246 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\") pod \"b85b7be5-bf2f-431c-89af-c24b14e2712b\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") "
Nov 03 05:57:59.621246 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621251    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\" (UniqueName: \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\") pod \"b85b7be5-bf2f-431c-89af-c24b14e2712b\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") "
Nov 03 05:57:59.621468 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621320    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\") pod \"b85b7be5-bf2f-431c-89af-c24b14e2712b\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") "
Nov 03 05:57:59.621468 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621339    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\") pod \"b85b7be5-bf2f-431c-89af-c24b14e2712b\" (UID: \"b85b7be5-bf2f-431c-89af-c24b14e2712b\") "
Nov 03 05:57:59.621648 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621625    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle" pod "b85b7be5-bf2f-431c-89af-c24b14e2712b" (UID: "b85b7be5-bf2f-431c-89af-c24b14e2712b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:57:59.621745 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path" pod "b85b7be5-bf2f-431c-89af-c24b14e2712b" (UID: "b85b7be5-bf2f-431c-89af-c24b14e2712b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:57:59.621818 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.621725    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle" pod "b85b7be5-bf2f-431c-89af-c24b14e2712b" (UID: "b85b7be5-bf2f-431c-89af-c24b14e2712b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 05:57:59.623475 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.623452    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng" pod "b85b7be5-bf2f-431c-89af-c24b14e2712b" (UID: "b85b7be5-bf2f-431c-89af-c24b14e2712b"). InnerVolumeSpecName "kube-api-access-cq9ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:57:59.722367 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.722275    4528 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-trusted-ca-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:57:59.722367 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.722309    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq9ng\" (UniqueName: \"kubernetes.io/projected/b85b7be5-bf2f-431c-89af-c24b14e2712b-kube-api-access-cq9ng\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:57:59.722367 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.722322    4528 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b85b7be5-bf2f-431c-89af-c24b14e2712b-service-ca-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:57:59.722367 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:57:59.722334    4528 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/b85b7be5-bf2f-431c-89af-c24b14e2712b-archives-path\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:58:00.428748 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:00.428717    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-cg65z-hbq8z"
Nov 03 05:58:00.428966 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:00.428710    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-cg65z-hbq8z" event={"ID":"b85b7be5-bf2f-431c-89af-c24b14e2712b","Type":"ContainerDied","Data":"a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15"}
Nov 03 05:58:00.428966 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:00.428840    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0967b92016258a3ce8d6c9b17ad5a0972409fee1e05f80888ac75a081912f15"
Nov 03 05:58:13.691780 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:58:13.691740    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:58:16.016542 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016504    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"]
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016787    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" containerName="insights-gathering"
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016813    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" containerName="insights-gathering"
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016824    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b85b7be5-bf2f-431c-89af-c24b14e2712b" containerName="insights-gathering"
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016829    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85b7be5-bf2f-431c-89af-c24b14e2712b" containerName="insights-gathering"
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016893    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fdd1c9-6616-4e37-9b39-ec6dc3d0435b" containerName="insights-gathering"
Nov 03 05:58:16.017045 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.016902    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85b7be5-bf2f-431c-89af-c24b14e2712b" containerName="insights-gathering"
Nov 03 05:58:16.020227 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.020207    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.034546 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.034525    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Nov 03 05:58:16.046059 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.046032    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Nov 03 05:58:16.047216 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.047190    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"]
Nov 03 05:58:16.058551 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.058530    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-bt5qj\""
Nov 03 05:58:16.159378 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.159341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.159586 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.159396    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6pnr\" (UniqueName: \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.159586 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.159509    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.260773 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.260735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6pnr\" (UniqueName: \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.260953 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.260812    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.260953 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.260854    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.261214 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.261198    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.261254 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.261230    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.284871 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.284787    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6pnr\" (UniqueName: \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.336194 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.336153    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:58:16.646949 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:16.646923    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"]
Nov 03 05:58:16.649497 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:16.649461    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c5311ad_fff7_4915_acc4_fd989d42c169.slice/crio-56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1 WatchSource:0}: Error finding container 56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1: Status 404 returned error can't find the container with id 56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1
Nov 03 05:58:17.472052 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.472022    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"]
Nov 03 05:58:17.475538 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.475511    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.489949 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.489914    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8" event={"ID":"0c5311ad-fff7-4915-acc4-fd989d42c169","Type":"ContainerStarted","Data":"56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1"}
Nov 03 05:58:17.501485 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.501464    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-gflwh\""
Nov 03 05:58:17.502875 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.502854    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"]
Nov 03 05:58:17.571895 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.571860    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.572072 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.571920    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w8gm\" (UniqueName: \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.572152 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.572081    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.578819 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.578779    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"]
Nov 03 05:58:17.582506 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.582486    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.600148 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.600095    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"]
Nov 03 05:58:17.601735 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.601700    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ngqgz\""
Nov 03 05:58:17.673070 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673028    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.673253 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673092    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.673253 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673125    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w8gm\" (UniqueName: \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.673253 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673172    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.673253 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673192    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.673253 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673219    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqv2z\" (UniqueName: \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.673655 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.673633    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.674040 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.674015    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.674447 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.674421    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"]
Nov 03 05:58:17.677802 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.677774    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.695432 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.695407    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"]
Nov 03 05:58:17.696701 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.696679    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-xpf9c\""
Nov 03 05:58:17.698971 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.698949    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w8gm\" (UniqueName: \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.770471 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.770396    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"]
Nov 03 05:58:17.774492 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774456    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.774771 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.774888 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774821    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqv2z\" (UniqueName: \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.774888 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774860    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.774994 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774892    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tzhq\" (UniqueName: \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.774994 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.774941    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.775088 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.775004    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.775407 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.775378    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"]
Nov 03 05:58:17.775499 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.775458    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.775946 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.775924    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.780178 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.780156    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.792147 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.792118    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"]
Nov 03 05:58:17.793356 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.793335    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:58:17.794617 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.794593    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-579hd\""
Nov 03 05:58:17.794714 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.794685    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-7wgkq\""
Nov 03 05:58:17.795940 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.795921    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"]
Nov 03 05:58:17.805666 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.805641    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqv2z\" (UniqueName: \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875637    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7pq2\" (UniqueName: \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875717    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875745    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875810    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb25r\" (UniqueName: \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875853    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.875871    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tzhq\" (UniqueName: \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.876514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.876810 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.876763    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.900746 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.900712    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:58:17.919846 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.919780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tzhq\" (UniqueName: \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:17.976953 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.976921    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.977120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.976961    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb25r\" (UniqueName: \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.977120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977017    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7pq2\" (UniqueName: \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.977120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977053    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.977120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977083    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.977120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977109    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.977725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977456    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.977725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977707    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.977725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977718    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:17.977906 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.977710    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:17.993584 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:17.993547    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:58:18.001063 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.001036    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7pq2\" (UniqueName: \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:18.001179 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.001032    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb25r\" (UniqueName: \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:18.094981 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.094934    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:58:18.102337 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.102296    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"]
Nov 03 05:58:18.103744 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.103726    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:58:18.106627 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:18.106595    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8694115d_d855_47ca_8d93_e70faac392e0.slice/crio-5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934 WatchSource:0}: Error finding container 5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934: Status 404 returned error can't find the container with id 5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934
Nov 03 05:58:18.205435 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.205398    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"]
Nov 03 05:58:18.210589 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:18.210555    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ad213b_715b_45e7_9c75_77c7b0d9929a.slice/crio-750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d WatchSource:0}: Error finding container 750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d: Status 404 returned error can't find the container with id 750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d
Nov 03 05:58:18.295503 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.295225    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"]
Nov 03 05:58:18.297670 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:18.297602    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11024810_ef22_4ca9_85dd_c1e95d5236cc.slice/crio-0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24 WatchSource:0}: Error finding container 0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24: Status 404 returned error can't find the container with id 0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24
Nov 03 05:58:18.397698 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.397663    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"]
Nov 03 05:58:18.401052 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:18.401027    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod643b9149_edda_4134_931a_000c0e2019a5.slice/crio-ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435 WatchSource:0}: Error finding container ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435: Status 404 returned error can't find the container with id ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435
Nov 03 05:58:18.409498 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.409434    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"]
Nov 03 05:58:18.412033 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:18.412006    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ef9b79_4e4f_4bad_a750_531885a87eac.slice/crio-9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab WatchSource:0}: Error finding container 9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab: Status 404 returned error can't find the container with id 9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab
Nov 03 05:58:18.497150 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.497098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc" event={"ID":"8694115d-d855-47ca-8d93-e70faac392e0","Type":"ContainerStarted","Data":"5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934"}
Nov 03 05:58:18.499577 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.499545    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerStarted","Data":"ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435"}
Nov 03 05:58:18.501140 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.501105    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq" event={"ID":"90ad213b-715b-45e7-9c75-77c7b0d9929a","Type":"ContainerStarted","Data":"750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d"}
Nov 03 05:58:18.502617 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.502589    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8" event={"ID":"e1ef9b79-4e4f-4bad-a750-531885a87eac","Type":"ContainerStarted","Data":"9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab"}
Nov 03 05:58:18.504819 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:18.504782    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs" event={"ID":"11024810-ef22-4ca9-85dd-c1e95d5236cc","Type":"ContainerStarted","Data":"0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24"}
Nov 03 05:58:19.365548 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.365494    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"]
Nov 03 05:58:19.369694 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.369667    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.405260 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.399987    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-qpd7b\""
Nov 03 05:58:19.405260 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.400763    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"]
Nov 03 05:58:19.490416 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.490361    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltq4\" (UniqueName: \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.490618 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.490460    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.490618 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.490487    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.591889 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.591854    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltq4\" (UniqueName: \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.592351 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.591914    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.592351 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.591933    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.592351 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.592320    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.592920 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.592893    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.625747 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.625643    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltq4\" (UniqueName: \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:19.692037 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:19.691999    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:58:20.032833 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:20.032719    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"]
Nov 03 05:58:20.519559 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:20.519518    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn" event={"ID":"a9cc6a50-401f-4bc3-847c-850da67f8fbc","Type":"ContainerStarted","Data":"cff8d0cc0f8cffd4ef745721af95b6d53b29523f8c6cabe2bb7b6440f123898b"}
Nov 03 05:58:22.299880 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.297758    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"]
Nov 03 05:58:22.301918 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.301893    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.335926 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.335839    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"]
Nov 03 05:58:22.423459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.423409    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.423637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.423483    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.423637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.423589    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.526637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.525059    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.526637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.525174    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.526637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.525268    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.526637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.525787    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.526637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.526535    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.560569 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.560481    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:22.632992 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:22.632676    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:58:23.011077 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:23.010338    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"]
Nov 03 05:58:23.013401 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:23.013370    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode475478b_56f6_42fa_b79e_0c5a9eff9f43.slice/crio-78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b WatchSource:0}: Error finding container 78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b: Status 404 returned error can't find the container with id 78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b
Nov 03 05:58:23.551418 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:23.551338    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r" event={"ID":"e475478b-56f6-42fa-b79e-0c5a9eff9f43","Type":"ContainerStarted","Data":"78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b"}
Nov 03 05:58:24.953720 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.950988    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"]
Nov 03 05:58:24.969783 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.969752    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:24.992942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.990212    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"]
Nov 03 05:58:24.992942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.990656    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 05:58:24.992942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.990969    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 05:58:24.992942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.990656    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 05:58:24.992942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:24.991192    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-j7v45\""
Nov 03 05:58:25.084578 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.083515    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"]
Nov 03 05:58:25.089437 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.089265    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.116724 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.116631    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"]
Nov 03 05:58:25.123866 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.123778    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-lsvqd\""
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162091    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162142    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162185    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162221    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162245    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162268    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162300    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162454    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.163067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.162481    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df2kt\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-kube-api-access-df2kt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.208067 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.205944    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"]
Nov 03 05:58:25.211730 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.210981    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.229129 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.227764    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"]
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265240    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265285    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265316    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265342    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265377    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.265460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265893    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265958    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnndd\" (UniqueName: \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.265988    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266038    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266193    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266328    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f8190f-d2db-4674-b2dc-7894853d97ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266406    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266450    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266481    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-424s9\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-kube-api-access-424s9\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266570    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266621    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df2kt\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-kube-api-access-df2kt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266651    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.266958 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.266694    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.267813 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.267008    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.267813 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.267399    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.273236 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.273149    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.273914 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.273862    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.274685 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.274662    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"]
Nov 03 05:58:25.279641 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.279620    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.299612 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.299581    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.301338 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.300518    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"]
Nov 03 05:58:25.305666 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.305639    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"]
Nov 03 05:58:25.311002 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.310681    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.313041 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.312974    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df2kt\" (UniqueName: \"kubernetes.io/projected/e8c4a3ba-5a15-40ac-823e-795c43744b3c-kube-api-access-df2kt\") pod \"maas-default-gateway-openshift-default-778b7b5c47-c4nqk\" (UID: \"e8c4a3ba-5a15-40ac-823e-795c43744b3c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.330726 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.330091    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"]
Nov 03 05:58:25.369619 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f8190f-d2db-4674-b2dc-7894853d97ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369632    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369663    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369697    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369725    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-424s9\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-kube-api-access-424s9\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fnnv\" (UniqueName: \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.369893 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369888    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369907    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369922    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369951    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnndd\" (UniqueName: \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.369985    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxhk\" (UniqueName: \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370013    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370055    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370421 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370381    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370979 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370418    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63f8190f-d2db-4674-b2dc-7894853d97ec-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370979 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370559    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.370979 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.370774    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.371138 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.371099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.371547 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.371521    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.372287 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.372210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.377508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.377481    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.381709 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.381684    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.399647 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.399618    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnndd\" (UniqueName: \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.400707 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.400651    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-424s9\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-kube-api-access-424s9\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.401655 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.401632    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63f8190f-d2db-4674-b2dc-7894853d97ec-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4\" (UID: \"63f8190f-d2db-4674-b2dc-7894853d97ec\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.410554 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.410526    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:25.471341 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.471256    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fnnv\" (UniqueName: \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.471509 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.471352    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxhk\" (UniqueName: \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.471509 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.471396    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.471509 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.471436    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.471671 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.471550    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.472874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.472164    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.472874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.472574    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.472874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.472640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.472874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.472830    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.473331 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.473308    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.521275 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.521200    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxhk\" (UniqueName: \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.524563 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.524457    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fnnv\" (UniqueName: \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.538411 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.538121    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:58:25.611554 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.611517    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:25.646272 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.646018    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:58:25.654412 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.654196    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:58:25.781319 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.781287    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"]
Nov 03 05:58:25.867210 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.867143    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"]
Nov 03 05:58:25.871929 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:25.871803    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44eb9f2_32f5_4887_9a5b_e7d8ce1130db.slice/crio-b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a WatchSource:0}: Error finding container b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a: Status 404 returned error can't find the container with id b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a
Nov 03 05:58:25.961065 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:25.960982    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"]
Nov 03 05:58:25.967073 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:25.967039    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c4a3ba_5a15_40ac_823e_795c43744b3c.slice/crio-4266d94d5078b05b78be67dd1f93399a474974c18ef28201783594da3244692f WatchSource:0}: Error finding container 4266d94d5078b05b78be67dd1f93399a474974c18ef28201783594da3244692f: Status 404 returned error can't find the container with id 4266d94d5078b05b78be67dd1f93399a474974c18ef28201783594da3244692f
Nov 03 05:58:26.002672 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.002635    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"]
Nov 03 05:58:26.041315 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.041283    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"]
Nov 03 05:58:26.047871 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:58:26.047842    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb44ce250_42ae_4dbd_92fc_af93c75a9e5d.slice/crio-24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de WatchSource:0}: Error finding container 24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de: Status 404 returned error can't find the container with id 24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de
Nov 03 05:58:26.613082 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.612941    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerStarted","Data":"2aeb54a28066c983cc136fdf1248be89cc96201d38ad29b605d1f5d3d3fecf90"}
Nov 03 05:58:26.635815 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.635692    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k" event={"ID":"b44ce250-42ae-4dbd-92fc-af93c75a9e5d","Type":"ContainerStarted","Data":"24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de"}
Nov 03 05:58:26.648697 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.648652    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4" event={"ID":"63f8190f-d2db-4674-b2dc-7894853d97ec","Type":"ContainerStarted","Data":"d34d1f4de3ee3dc096aa3932ed46d9a95acd5a9ece4b32d9fc307f0d858f30ed"}
Nov 03 05:58:26.675906 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.675860    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" event={"ID":"e8c4a3ba-5a15-40ac-823e-795c43744b3c","Type":"ContainerStarted","Data":"4266d94d5078b05b78be67dd1f93399a474974c18ef28201783594da3244692f"}
Nov 03 05:58:26.692355 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:26.692269    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8" event={"ID":"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db","Type":"ContainerStarted","Data":"b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a"}
Nov 03 05:58:35.804874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:35.804621    4528 generic.go:356] "Generic (PLEG): container finished" podID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerID="9dc5762365fc8858a0b71e873827ac69b4f51010a12303825e5829b2df0587c4" exitCode=0
Nov 03 05:58:35.804874 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:35.804742    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8" event={"ID":"0c5311ad-fff7-4915-acc4-fd989d42c169","Type":"ContainerDied","Data":"9dc5762365fc8858a0b71e873827ac69b4f51010a12303825e5829b2df0587c4"}
Nov 03 05:58:36.829484 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:36.829425    4528 generic.go:356] "Generic (PLEG): container finished" podID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerID="69e807dd37aaccb493a23dc01b9928ed491424d9dfbbf7e06b88aec61bce3c51" exitCode=0
Nov 03 05:58:36.829955 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:36.829600    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r" event={"ID":"e475478b-56f6-42fa-b79e-0c5a9eff9f43","Type":"ContainerDied","Data":"69e807dd37aaccb493a23dc01b9928ed491424d9dfbbf7e06b88aec61bce3c51"}
Nov 03 05:58:37.816267 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816224    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:37.816386 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816320    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:37.816386 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816364    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:37.816947 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816882    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:37.817015 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816957    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:37.817015 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:37.816999    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:58:38.861142 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.861104    4528 generic.go:356] "Generic (PLEG): container finished" podID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerID="b8e5f97c179254e39d752e8900515c7e11b31bc841d70ccef7701595849a78ae" exitCode=0
Nov 03 05:58:38.861591 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.861241    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8" event={"ID":"0c5311ad-fff7-4915-acc4-fd989d42c169","Type":"ContainerDied","Data":"b8e5f97c179254e39d752e8900515c7e11b31bc841d70ccef7701595849a78ae"}
Nov 03 05:58:38.866310 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.866125    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4" event={"ID":"63f8190f-d2db-4674-b2dc-7894853d97ec","Type":"ContainerStarted","Data":"65b205a4599243ad316333c6c88cd9693aa37c5ef603897d659d7839934f6122"}
Nov 03 05:58:38.869581 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.869545    4528 generic.go:356] "Generic (PLEG): container finished" podID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerID="d8d1683160ab8da21b4c64940705f1ab9b246258b5d8fa47be533248b2139a29" exitCode=0
Nov 03 05:58:38.869692 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.869619    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r" event={"ID":"e475478b-56f6-42fa-b79e-0c5a9eff9f43","Type":"ContainerDied","Data":"d8d1683160ab8da21b4c64940705f1ab9b246258b5d8fa47be533248b2139a29"}
Nov 03 05:58:38.872075 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.872016    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" event={"ID":"e8c4a3ba-5a15-40ac-823e-795c43744b3c","Type":"ContainerStarted","Data":"f093fdbe432791d0aa5949197047ad4092d88808fe7cbdb3d654ea29d310cb38"}
Nov 03 05:58:38.874258 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.874232    4528 generic.go:356] "Generic (PLEG): container finished" podID="8694115d-d855-47ca-8d93-e70faac392e0" containerID="00532c869343566726c0fd6c6188da8875c6d93e465a72628468d83bf361d2f9" exitCode=0
Nov 03 05:58:38.874397 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.874322    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc" event={"ID":"8694115d-d855-47ca-8d93-e70faac392e0","Type":"ContainerDied","Data":"00532c869343566726c0fd6c6188da8875c6d93e465a72628468d83bf361d2f9"}
Nov 03 05:58:38.876580 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.876549    4528 generic.go:356] "Generic (PLEG): container finished" podID="643b9149-edda-4134-931a-000c0e2019a5" containerID="43e5f1fd7c85dbfb1128ca0f810e5cdc0e4ede11d409ef8e7c4a46c52086f09c" exitCode=0
Nov 03 05:58:38.876694 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.876637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerDied","Data":"43e5f1fd7c85dbfb1128ca0f810e5cdc0e4ede11d409ef8e7c4a46c52086f09c"}
Nov 03 05:58:38.878769 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.878739    4528 generic.go:356] "Generic (PLEG): container finished" podID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerID="cac24d1fd7a9e83fa6b0bfa1732dc7e5b14f34c4349e12589bac3efa403003ea" exitCode=0
Nov 03 05:58:38.878918 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.878781    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq" event={"ID":"90ad213b-715b-45e7-9c75-77c7b0d9929a","Type":"ContainerDied","Data":"cac24d1fd7a9e83fa6b0bfa1732dc7e5b14f34c4349e12589bac3efa403003ea"}
Nov 03 05:58:38.955854 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:38.955787    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4" podStartSLOduration=1.94231159 podStartE2EDuration="13.955768958s" totalImagesPullingTime="12.013457368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:58:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:58:38.954285757 +0000 UTC m=+207.365682835" watchObservedRunningTime="2026-11-03 05:58:38.955768958 +0000 UTC m=+207.367166036"
Nov 03 05:58:39.080943 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.080880    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podStartSLOduration=3.240457321 podStartE2EDuration="15.080857295s" totalImagesPullingTime="11.840399974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:58:39.07854962 +0000 UTC m=+207.489946700" watchObservedRunningTime="2026-11-03 05:58:39.080857295 +0000 UTC m=+207.492254377"
Nov 03 05:58:39.410961 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.410928    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:39.416596 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.416568    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:39.612237 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.612197    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:39.616899 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.616873    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:39.887073 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.887036    4528 generic.go:356] "Generic (PLEG): container finished" podID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerID="949a2e0464e44c3a1a7e184f137e69559e01d90a1a7f241a8ddb49658cbab8e6" exitCode=0
Nov 03 05:58:39.887549 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.887153    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8" event={"ID":"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db","Type":"ContainerDied","Data":"949a2e0464e44c3a1a7e184f137e69559e01d90a1a7f241a8ddb49658cbab8e6"}
Nov 03 05:58:39.891027 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.890998    4528 generic.go:356] "Generic (PLEG): container finished" podID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerID="3a0dd48d5789003cf16c1e88a910c5a6ef19df7ed9b07d82e22ca03fc3743461" exitCode=0
Nov 03 05:58:39.891171 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.891096    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerDied","Data":"3a0dd48d5789003cf16c1e88a910c5a6ef19df7ed9b07d82e22ca03fc3743461"}
Nov 03 05:58:39.895151 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.894704    4528 generic.go:356] "Generic (PLEG): container finished" podID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerID="9c95f72619fa865ab6c40b190f882b1386079bd1d0b0ce3b84b0505c739280aa" exitCode=0
Nov 03 05:58:39.895151 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.895079    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k" event={"ID":"b44ce250-42ae-4dbd-92fc-af93c75a9e5d","Type":"ContainerDied","Data":"9c95f72619fa865ab6c40b190f882b1386079bd1d0b0ce3b84b0505c739280aa"}
Nov 03 05:58:39.904451 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.899002    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerID="352760dd64c98ccfc4faf802d1753672a71a913469d609c1ec64141991a54f7f" exitCode=0
Nov 03 05:58:39.904451 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.899102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8" event={"ID":"e1ef9b79-4e4f-4bad-a750-531885a87eac","Type":"ContainerDied","Data":"352760dd64c98ccfc4faf802d1753672a71a913469d609c1ec64141991a54f7f"}
Nov 03 05:58:39.909978 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.909571    4528 generic.go:356] "Generic (PLEG): container finished" podID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerID="5d1a3583478c89e95aca19e1a6e258732395174ee5a4065bba5be53c61a4c59f" exitCode=0
Nov 03 05:58:39.909978 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.909957    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs" event={"ID":"11024810-ef22-4ca9-85dd-c1e95d5236cc","Type":"ContainerDied","Data":"5d1a3583478c89e95aca19e1a6e258732395174ee5a4065bba5be53c61a4c59f"}
Nov 03 05:58:39.910551 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.910495    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:39.910653 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.910557    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:39.911258 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.911239    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4"
Nov 03 05:58:39.911686 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:39.911667    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 05:58:40.917946 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.917906    4528 generic.go:356] "Generic (PLEG): container finished" podID="8694115d-d855-47ca-8d93-e70faac392e0" containerID="ef0f13fbf16d69936ae3a32f16328c9865f593f2a40e09e12211fd418ef9edd3" exitCode=0
Nov 03 05:58:40.918456 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.918015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc" event={"ID":"8694115d-d855-47ca-8d93-e70faac392e0","Type":"ContainerDied","Data":"ef0f13fbf16d69936ae3a32f16328c9865f593f2a40e09e12211fd418ef9edd3"}
Nov 03 05:58:40.921491 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.921462    4528 generic.go:356] "Generic (PLEG): container finished" podID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerID="b740795ac9e0dd40650466a84e707a28fa7e488f4359e9868a2e26d2f2f021cf" exitCode=0
Nov 03 05:58:40.921593 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.921573    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerDied","Data":"b740795ac9e0dd40650466a84e707a28fa7e488f4359e9868a2e26d2f2f021cf"}
Nov 03 05:58:40.924443 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.924415    4528 generic.go:356] "Generic (PLEG): container finished" podID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerID="7bcbd2210ce584b6917db49879185a10922c02e51a20a612ec7fad5ff44fadba" exitCode=0
Nov 03 05:58:40.924551 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.924489    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k" event={"ID":"b44ce250-42ae-4dbd-92fc-af93c75a9e5d","Type":"ContainerDied","Data":"7bcbd2210ce584b6917db49879185a10922c02e51a20a612ec7fad5ff44fadba"}
Nov 03 05:58:40.928541 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.928515    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerID="c2520ceaa228b15e851fcc8ca3d1a126938db72e52144b628822eb37480e2317" exitCode=0
Nov 03 05:58:40.928651 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.928598    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8" event={"ID":"e1ef9b79-4e4f-4bad-a750-531885a87eac","Type":"ContainerDied","Data":"c2520ceaa228b15e851fcc8ca3d1a126938db72e52144b628822eb37480e2317"}
Nov 03 05:58:40.930706 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.930674    4528 generic.go:356] "Generic (PLEG): container finished" podID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerID="c2e7b37633873f5e39139db71bdad19ace4b2cdbf3c49b12cc5fa0dd508baa16" exitCode=0
Nov 03 05:58:40.930821 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.930773    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn" event={"ID":"a9cc6a50-401f-4bc3-847c-850da67f8fbc","Type":"ContainerDied","Data":"c2e7b37633873f5e39139db71bdad19ace4b2cdbf3c49b12cc5fa0dd508baa16"}
Nov 03 05:58:40.933680 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.933656    4528 generic.go:356] "Generic (PLEG): container finished" podID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerID="65dec6037a23ad9464fa0fc83681f84029042d365eca2310d3510f64bc86edf0" exitCode=0
Nov 03 05:58:40.933821 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.933757    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs" event={"ID":"11024810-ef22-4ca9-85dd-c1e95d5236cc","Type":"ContainerDied","Data":"65dec6037a23ad9464fa0fc83681f84029042d365eca2310d3510f64bc86edf0"}
Nov 03 05:58:40.936829 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.936802    4528 generic.go:356] "Generic (PLEG): container finished" podID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerID="0c127e207e618b4c5c8848748ebeea6cc0cc5dc7f5eac986c73effffc0ff038c" exitCode=0
Nov 03 05:58:40.937871 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:40.937847    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8" event={"ID":"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db","Type":"ContainerDied","Data":"0c127e207e618b4c5c8848748ebeea6cc0cc5dc7f5eac986c73effffc0ff038c"}
Nov 03 05:58:42.968312 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:42.968265    4528 generic.go:356] "Generic (PLEG): container finished" podID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerID="037addb52daccb84552ab9278578d95de7def644d489a4f4be33ea5e1de58266" exitCode=0
Nov 03 05:58:42.968869 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:42.968384    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq" event={"ID":"90ad213b-715b-45e7-9c75-77c7b0d9929a","Type":"ContainerDied","Data":"037addb52daccb84552ab9278578d95de7def644d489a4f4be33ea5e1de58266"}
Nov 03 05:58:43.761977 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:58:43.761936    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:58:49.077675 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:49.076569    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerStarted","Data":"a0bdad78227c2ba91d229d6bdf47e3122f147ef9e44da6dcf60ead71b05eafcf"}
Nov 03 05:58:49.094456 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:49.091539    4528 generic.go:356] "Generic (PLEG): container finished" podID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerID="348d0b9ece7d5fdf03134143bef7cc247215a96ef6c0c197ebbab0acd4d5614a" exitCode=0
Nov 03 05:58:49.094456 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:49.091675    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn" event={"ID":"a9cc6a50-401f-4bc3-847c-850da67f8fbc","Type":"ContainerDied","Data":"348d0b9ece7d5fdf03134143bef7cc247215a96ef6c0c197ebbab0acd4d5614a"}
Nov 03 05:58:50.133352 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:50.133313    4528 generic.go:356] "Generic (PLEG): container finished" podID="643b9149-edda-4134-931a-000c0e2019a5" containerID="a0bdad78227c2ba91d229d6bdf47e3122f147ef9e44da6dcf60ead71b05eafcf" exitCode=0
Nov 03 05:58:50.133861 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:58:50.133408    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerDied","Data":"a0bdad78227c2ba91d229d6bdf47e3122f147ef9e44da6dcf60ead71b05eafcf"}
Nov 03 05:59:00.273052 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:00.272727    4528 generic.go:356] "Generic (PLEG): container finished" podID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerID="5164a9aa2b8ef486738f9bfd7484b9cbff618017dd6b6432d47fd7f3dd294224" exitCode=0
Nov 03 05:59:00.273052 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:00.272889    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8" event={"ID":"0c5311ad-fff7-4915-acc4-fd989d42c169","Type":"ContainerDied","Data":"5164a9aa2b8ef486738f9bfd7484b9cbff618017dd6b6432d47fd7f3dd294224"}
Nov 03 05:59:01.289452 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.288624    4528 generic.go:356] "Generic (PLEG): container finished" podID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerID="103d1c5c3e6da9ef7030acf3bb29a92010d2560c23f15181542de43bf941dedd" exitCode=0
Nov 03 05:59:01.289452 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.289389    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r" event={"ID":"e475478b-56f6-42fa-b79e-0c5a9eff9f43","Type":"ContainerDied","Data":"103d1c5c3e6da9ef7030acf3bb29a92010d2560c23f15181542de43bf941dedd"}
Nov 03 05:59:01.493855 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.493509    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:59:01.576044 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.575339    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\") pod \"0c5311ad-fff7-4915-acc4-fd989d42c169\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") "
Nov 03 05:59:01.576044 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.575400    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\" (UniqueName: \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\") pod \"0c5311ad-fff7-4915-acc4-fd989d42c169\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") "
Nov 03 05:59:01.576044 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.575495    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\") pod \"0c5311ad-fff7-4915-acc4-fd989d42c169\" (UID: \"0c5311ad-fff7-4915-acc4-fd989d42c169\") "
Nov 03 05:59:01.577333 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.576821    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle" pod "0c5311ad-fff7-4915-acc4-fd989d42c169" (UID: "0c5311ad-fff7-4915-acc4-fd989d42c169"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:01.584832 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.584802    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util" pod "0c5311ad-fff7-4915-acc4-fd989d42c169" (UID: "0c5311ad-fff7-4915-acc4-fd989d42c169"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:01.591679 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.591642    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr" pod "0c5311ad-fff7-4915-acc4-fd989d42c169" (UID: "0c5311ad-fff7-4915-acc4-fd989d42c169"). InnerVolumeSpecName "kube-api-access-m6pnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:01.677363 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.677273    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:01.677363 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.677313    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0c5311ad-fff7-4915-acc4-fd989d42c169-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:01.677363 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:01.677331    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6pnr\" (UniqueName: \"kubernetes.io/projected/0c5311ad-fff7-4915-acc4-fd989d42c169-kube-api-access-m6pnr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:02.312863 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.312212    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8"
Nov 03 05:59:02.312863 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.312282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71866k8" event={"ID":"0c5311ad-fff7-4915-acc4-fd989d42c169","Type":"ContainerDied","Data":"56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1"}
Nov 03 05:59:02.312863 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.312334    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56f14322e83c4e505e1a9be46b9cd098639e54243a62251645ec778cc005b9a1"
Nov 03 05:59:02.612120 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.612082    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:59:02.690090 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.690054    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\") pod \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") "
Nov 03 05:59:02.690256 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.690105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\") pod \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") "
Nov 03 05:59:02.690256 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.690153    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\") pod \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\" (UID: \"e475478b-56f6-42fa-b79e-0c5a9eff9f43\") "
Nov 03 05:59:02.690718 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.690690    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle" pod "e475478b-56f6-42fa-b79e-0c5a9eff9f43" (UID: "e475478b-56f6-42fa-b79e-0c5a9eff9f43"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:02.692719 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.692694    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp" pod "e475478b-56f6-42fa-b79e-0c5a9eff9f43" (UID: "e475478b-56f6-42fa-b79e-0c5a9eff9f43"). InnerVolumeSpecName "kube-api-access-hwdlp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:02.696803 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.696759    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util" pod "e475478b-56f6-42fa-b79e-0c5a9eff9f43" (UID: "e475478b-56f6-42fa-b79e-0c5a9eff9f43"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:02.791602 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.791513    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:02.791602 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.791549    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e475478b-56f6-42fa-b79e-0c5a9eff9f43-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:02.791602 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:02.791564    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwdlp\" (UniqueName: \"kubernetes.io/projected/e475478b-56f6-42fa-b79e-0c5a9eff9f43-kube-api-access-hwdlp\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:03.320542 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.320504    4528 generic.go:356] "Generic (PLEG): container finished" podID="643b9149-edda-4134-931a-000c0e2019a5" containerID="afaed19b39fb17bd234fcdb8b1d1cbd5228c0605484bc3046871bda946e9e5ce" exitCode=0
Nov 03 05:59:03.321026 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.320588    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerDied","Data":"afaed19b39fb17bd234fcdb8b1d1cbd5228c0605484bc3046871bda946e9e5ce"}
Nov 03 05:59:03.322921 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.322891    4528 generic.go:356] "Generic (PLEG): container finished" podID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerID="9e92b6608699c7e55f69ccff9e4431facff6f245080410b11dc521f597a1aa8f" exitCode=0
Nov 03 05:59:03.323054 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.322982    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn" event={"ID":"a9cc6a50-401f-4bc3-847c-850da67f8fbc","Type":"ContainerDied","Data":"9e92b6608699c7e55f69ccff9e4431facff6f245080410b11dc521f597a1aa8f"}
Nov 03 05:59:03.325275 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.325254    4528 generic.go:356] "Generic (PLEG): container finished" podID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerID="0237aa61b3a1814cfa0fc07b988a6b5f73d6ebd39d0d67dfc37c93269c7c6b60" exitCode=0
Nov 03 05:59:03.325370 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.325316    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs" event={"ID":"11024810-ef22-4ca9-85dd-c1e95d5236cc","Type":"ContainerDied","Data":"0237aa61b3a1814cfa0fc07b988a6b5f73d6ebd39d0d67dfc37c93269c7c6b60"}
Nov 03 05:59:03.327425 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.327363    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r"
Nov 03 05:59:03.327425 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.327393    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71ssq4r" event={"ID":"e475478b-56f6-42fa-b79e-0c5a9eff9f43","Type":"ContainerDied","Data":"78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b"}
Nov 03 05:59:03.327587 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.327431    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78896b0ba1911f92d7d1635d98f67ee835bfc6bc66d90827ca631afac0d8b40b"
Nov 03 05:59:03.330619 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.330596    4528 generic.go:356] "Generic (PLEG): container finished" podID="8694115d-d855-47ca-8d93-e70faac392e0" containerID="d4e4632d4dfa640103f1f6be907c8dbf6a127b59a7e5d620dc423c8d6247db1f" exitCode=0
Nov 03 05:59:03.330708 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.330661    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc" event={"ID":"8694115d-d855-47ca-8d93-e70faac392e0","Type":"ContainerDied","Data":"d4e4632d4dfa640103f1f6be907c8dbf6a127b59a7e5d620dc423c8d6247db1f"}
Nov 03 05:59:03.333064 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.333036    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerStarted","Data":"22037e39fdec33418db518ca84b44ee9661cafc5ec47821dc4c605504a30b9c2"}
Nov 03 05:59:03.454460 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:03.454392    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" podStartSLOduration=2.314947736 podStartE2EDuration="38.454371055s" totalImagesPullingTime="36.139423319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:58:25 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:03.453697015 +0000 UTC m=+231.865094094" watchObservedRunningTime="2026-11-03 05:59:03.454371055 +0000 UTC m=+231.865768134"
Nov 03 05:59:04.339455 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.339415    4528 generic.go:356] "Generic (PLEG): container finished" podID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerID="2dff77c79aefb3614e3d9b130828ea1e84b7e5f318787d4bebf2f9386cadd7b4" exitCode=0
Nov 03 05:59:04.339897 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.339503    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8" event={"ID":"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db","Type":"ContainerDied","Data":"2dff77c79aefb3614e3d9b130828ea1e84b7e5f318787d4bebf2f9386cadd7b4"}
Nov 03 05:59:04.341492 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.341470    4528 generic.go:356] "Generic (PLEG): container finished" podID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerID="22037e39fdec33418db518ca84b44ee9661cafc5ec47821dc4c605504a30b9c2" exitCode=0
Nov 03 05:59:04.341594 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.341543    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerDied","Data":"22037e39fdec33418db518ca84b44ee9661cafc5ec47821dc4c605504a30b9c2"}
Nov 03 05:59:04.343492 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.343475    4528 generic.go:356] "Generic (PLEG): container finished" podID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerID="aab4f72e37f265558c3b49f64b199f97fc411d345f7b4015065a9684274d50f7" exitCode=0
Nov 03 05:59:04.343606 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.343549    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k" event={"ID":"b44ce250-42ae-4dbd-92fc-af93c75a9e5d","Type":"ContainerDied","Data":"aab4f72e37f265558c3b49f64b199f97fc411d345f7b4015065a9684274d50f7"}
Nov 03 05:59:04.345523 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.345498    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerID="371eb467d98cf6ccf4e546ee49983cacdc7c22e9097db780424eb11f55b227fa" exitCode=0
Nov 03 05:59:04.345629 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.345585    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8" event={"ID":"e1ef9b79-4e4f-4bad-a750-531885a87eac","Type":"ContainerDied","Data":"371eb467d98cf6ccf4e546ee49983cacdc7c22e9097db780424eb11f55b227fa"}
Nov 03 05:59:04.478339 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.478235    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:59:04.573027 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.573002    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:59:04.577709 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.577689    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:59:04.581470 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.581452    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:59:04.607474 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.607385    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\") pod \"643b9149-edda-4134-931a-000c0e2019a5\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") "
Nov 03 05:59:04.607649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.607488    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\") pod \"643b9149-edda-4134-931a-000c0e2019a5\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") "
Nov 03 05:59:04.607649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.607618    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\" (UniqueName: \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\") pod \"643b9149-edda-4134-931a-000c0e2019a5\" (UID: \"643b9149-edda-4134-931a-000c0e2019a5\") "
Nov 03 05:59:04.607851 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.607821    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle" pod "643b9149-edda-4134-931a-000c0e2019a5" (UID: "643b9149-edda-4134-931a-000c0e2019a5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.612189 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.612159    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r" pod "643b9149-edda-4134-931a-000c0e2019a5" (UID: "643b9149-edda-4134-931a-000c0e2019a5"). InnerVolumeSpecName "kube-api-access-rb25r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:04.615037 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.615004    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util" pod "643b9149-edda-4134-931a-000c0e2019a5" (UID: "643b9149-edda-4134-931a-000c0e2019a5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.708389 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708358    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\") pod \"8694115d-d855-47ca-8d93-e70faac392e0\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") "
Nov 03 05:59:04.708513 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708425    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\") pod \"11024810-ef22-4ca9-85dd-c1e95d5236cc\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") "
Nov 03 05:59:04.708513 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708450    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\") pod \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") "
Nov 03 05:59:04.708513 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708487    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\") pod \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") "
Nov 03 05:59:04.708638 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708542    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\" (UniqueName: \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\") pod \"8694115d-d855-47ca-8d93-e70faac392e0\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") "
Nov 03 05:59:04.708638 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708586    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\") pod \"8694115d-d855-47ca-8d93-e70faac392e0\" (UID: \"8694115d-d855-47ca-8d93-e70faac392e0\") "
Nov 03 05:59:04.708638 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708630    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\") pod \"11024810-ef22-4ca9-85dd-c1e95d5236cc\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") "
Nov 03 05:59:04.708786 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708671    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\" (UniqueName: \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\") pod \"11024810-ef22-4ca9-85dd-c1e95d5236cc\" (UID: \"11024810-ef22-4ca9-85dd-c1e95d5236cc\") "
Nov 03 05:59:04.708786 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.708706    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\" (UniqueName: \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\") pod \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\" (UID: \"a9cc6a50-401f-4bc3-847c-850da67f8fbc\") "
Nov 03 05:59:04.709073 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.709050    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb25r\" (UniqueName: \"kubernetes.io/projected/643b9149-edda-4134-931a-000c0e2019a5-kube-api-access-rb25r\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.709591    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle" pod "11024810-ef22-4ca9-85dd-c1e95d5236cc" (UID: "11024810-ef22-4ca9-85dd-c1e95d5236cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.710209    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.710237    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/643b9149-edda-4134-931a-000c0e2019a5-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.711485    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle" pod "a9cc6a50-401f-4bc3-847c-850da67f8fbc" (UID: "a9cc6a50-401f-4bc3-847c-850da67f8fbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.711645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle" pod "8694115d-d855-47ca-8d93-e70faac392e0" (UID: "8694115d-d855-47ca-8d93-e70faac392e0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.712915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq" pod "11024810-ef22-4ca9-85dd-c1e95d5236cc" (UID: "11024810-ef22-4ca9-85dd-c1e95d5236cc"). InnerVolumeSpecName "kube-api-access-8tzhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.713631    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm" pod "8694115d-d855-47ca-8d93-e70faac392e0" (UID: "8694115d-d855-47ca-8d93-e70faac392e0"). InnerVolumeSpecName "kube-api-access-6w8gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:04.715766 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.714358    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4" pod "a9cc6a50-401f-4bc3-847c-850da67f8fbc" (UID: "a9cc6a50-401f-4bc3-847c-850da67f8fbc"). InnerVolumeSpecName "kube-api-access-zltq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:04.718248 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.718222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util" pod "a9cc6a50-401f-4bc3-847c-850da67f8fbc" (UID: "a9cc6a50-401f-4bc3-847c-850da67f8fbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.720736 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.720712    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util" pod "11024810-ef22-4ca9-85dd-c1e95d5236cc" (UID: "11024810-ef22-4ca9-85dd-c1e95d5236cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.721942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.721917    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util" pod "8694115d-d855-47ca-8d93-e70faac392e0" (UID: "8694115d-d855-47ca-8d93-e70faac392e0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:04.811657 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811615    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811657 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811650    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tzhq\" (UniqueName: \"kubernetes.io/projected/11024810-ef22-4ca9-85dd-c1e95d5236cc-kube-api-access-8tzhq\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811669    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zltq4\" (UniqueName: \"kubernetes.io/projected/a9cc6a50-401f-4bc3-847c-850da67f8fbc-kube-api-access-zltq4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811686    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811700    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/11024810-ef22-4ca9-85dd-c1e95d5236cc-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811709    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811719    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a9cc6a50-401f-4bc3-847c-850da67f8fbc-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811730    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w8gm\" (UniqueName: \"kubernetes.io/projected/8694115d-d855-47ca-8d93-e70faac392e0-kube-api-access-6w8gm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:04.811908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:04.811744    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8694115d-d855-47ca-8d93-e70faac392e0-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.352761 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.352723    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc" event={"ID":"8694115d-d855-47ca-8d93-e70faac392e0","Type":"ContainerDied","Data":"5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934"}
Nov 03 05:59:05.353353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.352772    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d5b2cfc10a60d078ce78d49b837caaa7a2c1b49d3a074296bc96d3c7acdd934"
Nov 03 05:59:05.353353 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.352737    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c69q6mc"
Nov 03 05:59:05.354777 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.354750    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv"
Nov 03 05:59:05.354908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.354750    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e64458htv" event={"ID":"643b9149-edda-4134-931a-000c0e2019a5","Type":"ContainerDied","Data":"ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435"}
Nov 03 05:59:05.354908 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.354842    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3a686e454ea025dfea67068863d334a8ab76c76fdda4f01b3b5b6695e74435"
Nov 03 05:59:05.357173 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.357150    4528 generic.go:356] "Generic (PLEG): container finished" podID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerID="4666b57fe2d40cd45a8217dc25de771a56be84f8485183c345261f55eb298a8d" exitCode=0
Nov 03 05:59:05.357283 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.357251    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq" event={"ID":"90ad213b-715b-45e7-9c75-77c7b0d9929a","Type":"ContainerDied","Data":"4666b57fe2d40cd45a8217dc25de771a56be84f8485183c345261f55eb298a8d"}
Nov 03 05:59:05.359200 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.359176    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn"
Nov 03 05:59:05.359304 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.359180    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cldzfn" event={"ID":"a9cc6a50-401f-4bc3-847c-850da67f8fbc","Type":"ContainerDied","Data":"cff8d0cc0f8cffd4ef745721af95b6d53b29523f8c6cabe2bb7b6440f123898b"}
Nov 03 05:59:05.359304 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.359289    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff8d0cc0f8cffd4ef745721af95b6d53b29523f8c6cabe2bb7b6440f123898b"
Nov 03 05:59:05.361358 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.361318    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs"
Nov 03 05:59:05.361453 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.361371    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cbs6qs" event={"ID":"11024810-ef22-4ca9-85dd-c1e95d5236cc","Type":"ContainerDied","Data":"0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24"}
Nov 03 05:59:05.361453 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.361445    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0270dd04140ba90d4cccb7ef1075aae5449552934170d2157a19375bfd50fe24"
Nov 03 05:59:05.535388 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.529139    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:59:05.535388 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.532411    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:59:05.560534 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.560508    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:59:05.569393 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.569368    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:59:05.619305 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619222    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\") pod \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") "
Nov 03 05:59:05.619305 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619303    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\" (UniqueName: \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\") pod \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") "
Nov 03 05:59:05.619533 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619326    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\") pod \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") "
Nov 03 05:59:05.619533 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619353    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\" (UniqueName: \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\") pod \"e1ef9b79-4e4f-4bad-a750-531885a87eac\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") "
Nov 03 05:59:05.619533 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619382    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\") pod \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\" (UID: \"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db\") "
Nov 03 05:59:05.619533 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619424    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\") pod \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") "
Nov 03 05:59:05.619533 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619463    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\") pod \"e1ef9b79-4e4f-4bad-a750-531885a87eac\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") "
Nov 03 05:59:05.619845 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619557    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\" (UniqueName: \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\") pod \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\" (UID: \"b44ce250-42ae-4dbd-92fc-af93c75a9e5d\") "
Nov 03 05:59:05.619845 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.619584    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\") pod \"e1ef9b79-4e4f-4bad-a750-531885a87eac\" (UID: \"e1ef9b79-4e4f-4bad-a750-531885a87eac\") "
Nov 03 05:59:05.620508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.620299    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle" pod "b44ce250-42ae-4dbd-92fc-af93c75a9e5d" (UID: "b44ce250-42ae-4dbd-92fc-af93c75a9e5d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.620508 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.620454    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle" pod "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" (UID: "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.620681 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.620585    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle" pod "e1ef9b79-4e4f-4bad-a750-531885a87eac" (UID: "e1ef9b79-4e4f-4bad-a750-531885a87eac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.622462 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.622153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2" pod "e1ef9b79-4e4f-4bad-a750-531885a87eac" (UID: "e1ef9b79-4e4f-4bad-a750-531885a87eac"). InnerVolumeSpecName "kube-api-access-j7pq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:05.623769 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.623693    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv" pod "b44ce250-42ae-4dbd-92fc-af93c75a9e5d" (UID: "b44ce250-42ae-4dbd-92fc-af93c75a9e5d"). InnerVolumeSpecName "kube-api-access-2fnnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:05.623769 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.623727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd" pod "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" (UID: "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db"). InnerVolumeSpecName "kube-api-access-tnndd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:05.628205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.628173    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util" pod "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" (UID: "d44eb9f2-32f5-4887-9a5b-e7d8ce1130db"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.628864 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.628833    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util" pod "b44ce250-42ae-4dbd-92fc-af93c75a9e5d" (UID: "b44ce250-42ae-4dbd-92fc-af93c75a9e5d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.629050 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.629031    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util" pod "e1ef9b79-4e4f-4bad-a750-531885a87eac" (UID: "e1ef9b79-4e4f-4bad-a750-531885a87eac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.720972 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.720939    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\") pod \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") "
Nov 03 05:59:05.721168 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.720984    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\" (UniqueName: \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\") pod \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") "
Nov 03 05:59:05.721168 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721006    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\") pod \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\" (UID: \"191ad00c-8d5b-4395-a9f6-647fdcedb1ac\") "
Nov 03 05:59:05.721321 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721299    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7pq2\" (UniqueName: \"kubernetes.io/projected/e1ef9b79-4e4f-4bad-a750-531885a87eac-kube-api-access-j7pq2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721382 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721327    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721382 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721342    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721382 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721355    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721382 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721368    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fnnv\" (UniqueName: \"kubernetes.io/projected/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-kube-api-access-2fnnv\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721575 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721383    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e1ef9b79-4e4f-4bad-a750-531885a87eac-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721575 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721396    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721575 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721409    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnndd\" (UniqueName: \"kubernetes.io/projected/d44eb9f2-32f5-4887-9a5b-e7d8ce1130db-kube-api-access-tnndd\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721575 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721422    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b44ce250-42ae-4dbd-92fc-af93c75a9e5d-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.721776 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.721561    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle" pod "191ad00c-8d5b-4395-a9f6-647fdcedb1ac" (UID: "191ad00c-8d5b-4395-a9f6-647fdcedb1ac"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.723217 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.723192    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk" pod "191ad00c-8d5b-4395-a9f6-647fdcedb1ac" (UID: "191ad00c-8d5b-4395-a9f6-647fdcedb1ac"). InnerVolumeSpecName "kube-api-access-zxxhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:05.727535 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.727502    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util" pod "191ad00c-8d5b-4395-a9f6-647fdcedb1ac" (UID: "191ad00c-8d5b-4395-a9f6-647fdcedb1ac"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:05.822409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.822366    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.822409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.822406    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxxhk\" (UniqueName: \"kubernetes.io/projected/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-kube-api-access-zxxhk\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:05.822409 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:05.822418    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/191ad00c-8d5b-4395-a9f6-647fdcedb1ac-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:06.082675 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.082639    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"]
Nov 03 05:59:06.082994 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.082980    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="pull"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.082998    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="pull"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083012    4528 cpu_manager.go:414] "containerMap: removing container" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="util"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083017    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="util"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083024    4528 cpu_manager.go:414] "containerMap: removing container" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="pull"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083029    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="pull"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083036    4528 cpu_manager.go:414] "containerMap: removing container" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="pull"
Nov 03 05:59:06.083048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083040    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083052    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083057    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083063    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083068    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083076    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083083    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083094    4528 cpu_manager.go:414] "containerMap: removing container" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083098    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083104    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083109    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083115    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083118    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083124    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083129    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083136    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083140    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083149    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083156    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083167    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083172    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083178    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083182    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083188    4528 cpu_manager.go:414] "containerMap: removing container" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083192    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="extract"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083200    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083204    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="pull"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083212    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083216    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083221    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083226    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083236    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083242    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083252    4528 cpu_manager.go:414] "containerMap: removing container" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="util"
Nov 03 05:59:06.083245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083257    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083263    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083267    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083275    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083279    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083285    4528 cpu_manager.go:414] "containerMap: removing container" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083289    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083293    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083297    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083304    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083308    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083318    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083324    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="pull"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083331    4528 cpu_manager.go:414] "containerMap: removing container" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083335    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083341    4528 cpu_manager.go:414] "containerMap: removing container" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083345    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083352    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083356    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="util"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083413    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b44ce250-42ae-4dbd-92fc-af93c75a9e5d" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083421    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d44eb9f2-32f5-4887-9a5b-e7d8ce1130db" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083428    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8694115d-d855-47ca-8d93-e70faac392e0" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083432    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e475478b-56f6-42fa-b79e-0c5a9eff9f43" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083437    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c5311ad-fff7-4915-acc4-fd989d42c169" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083442    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9cc6a50-401f-4bc3-847c-850da67f8fbc" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083447    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="643b9149-edda-4134-931a-000c0e2019a5" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083453    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1ef9b79-4e4f-4bad-a750-531885a87eac" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083459    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="11024810-ef22-4ca9-85dd-c1e95d5236cc" containerName="extract"
Nov 03 05:59:06.084205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.083464    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="191ad00c-8d5b-4395-a9f6-647fdcedb1ac" containerName="extract"
Nov 03 05:59:06.086350 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.086331    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.125307 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.125273    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 05:59:06.125307 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.125310    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 05:59:06.125520 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.125326    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 05:59:06.125586 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.125567    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 05:59:06.126009 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.125995    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-5996c\""
Nov 03 05:59:06.132756 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.132740    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 05:59:06.186987 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.186954    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"]
Nov 03 05:59:06.226742 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.226701    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p8wk\" (UniqueName: \"kubernetes.io/projected/680c9d72-4f8d-49c0-95cd-d25aab3710ef-kube-api-access-8p8wk\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.226951 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.226743    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.226951 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.226776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.226951 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.226936    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.227095 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.226995    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.227095 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.227029    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.327880 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.327836    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.327913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p8wk\" (UniqueName: \"kubernetes.io/projected/680c9d72-4f8d-49c0-95cd-d25aab3710ef-kube-api-access-8p8wk\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.327941    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.327971    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.328002    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328087 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.328038    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.328543 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.328513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-manifests\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.330333 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.330305    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/680c9d72-4f8d-49c0-95cd-d25aab3710ef-tmp\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.330551 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.330530    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-webhook-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.330626 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.330611    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-metrics-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.330685 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.330640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/680c9d72-4f8d-49c0-95cd-d25aab3710ef-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.351285 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.351221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p8wk\" (UniqueName: \"kubernetes.io/projected/680c9d72-4f8d-49c0-95cd-d25aab3710ef-kube-api-access-8p8wk\") pod \"opendatahub-operator-controller-manager-54b885cf97-5hk8f\" (UID: \"680c9d72-4f8d-49c0-95cd-d25aab3710ef\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.367830 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.367784    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx"
Nov 03 05:59:06.368245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.367785    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6pjxpx" event={"ID":"191ad00c-8d5b-4395-a9f6-647fdcedb1ac","Type":"ContainerDied","Data":"2aeb54a28066c983cc136fdf1248be89cc96201d38ad29b605d1f5d3d3fecf90"}
Nov 03 05:59:06.368245 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.367928    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2aeb54a28066c983cc136fdf1248be89cc96201d38ad29b605d1f5d3d3fecf90"
Nov 03 05:59:06.370545 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.369977    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k" event={"ID":"b44ce250-42ae-4dbd-92fc-af93c75a9e5d","Type":"ContainerDied","Data":"24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de"}
Nov 03 05:59:06.370545 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.370028    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24f19a9d0ac5da1060ee5cde170a1337d72deccc7f90a3441fc7f9dfe3b456de"
Nov 03 05:59:06.370545 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.369986    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7dkm6k"
Nov 03 05:59:06.372361 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.372334    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8" event={"ID":"e1ef9b79-4e4f-4bad-a750-531885a87eac","Type":"ContainerDied","Data":"9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab"}
Nov 03 05:59:06.372457 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.372382    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bc0f763a334156f21b00f58c8eff9e4ae3f5512a34fe6e13e2a6ce2c322f9ab"
Nov 03 05:59:06.372499 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.372474    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ajj6q8"
Nov 03 05:59:06.374566 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.374540    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8"
Nov 03 05:59:06.374653 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.374583    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cqwsw8" event={"ID":"d44eb9f2-32f5-4887-9a5b-e7d8ce1130db","Type":"ContainerDied","Data":"b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a"}
Nov 03 05:59:06.374653 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.374618    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e99daa071e0c8d291775affafbb8aede9e61cb4b99fdf3576dc4ee9d2d366a"
Nov 03 05:59:06.400266 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.400235    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:06.518085 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.518061    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:59:06.630379 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.630279    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\") pod \"90ad213b-715b-45e7-9c75-77c7b0d9929a\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") "
Nov 03 05:59:06.630566 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.630424    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\" (UniqueName: \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\") pod \"90ad213b-715b-45e7-9c75-77c7b0d9929a\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") "
Nov 03 05:59:06.630566 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.630456    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\") pod \"90ad213b-715b-45e7-9c75-77c7b0d9929a\" (UID: \"90ad213b-715b-45e7-9c75-77c7b0d9929a\") "
Nov 03 05:59:06.630926 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.630893    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle" pod "90ad213b-715b-45e7-9c75-77c7b0d9929a" (UID: "90ad213b-715b-45e7-9c75-77c7b0d9929a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:06.632605 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.632581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z" pod "90ad213b-715b-45e7-9c75-77c7b0d9929a" (UID: "90ad213b-715b-45e7-9c75-77c7b0d9929a"). InnerVolumeSpecName "kube-api-access-rqv2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 05:59:06.636217 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.636197    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util" pod "90ad213b-715b-45e7-9c75-77c7b0d9929a" (UID: "90ad213b-715b-45e7-9c75-77c7b0d9929a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 05:59:06.731892 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.731856    4528 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-bundle\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:06.731892 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.731884    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqv2z\" (UniqueName: \"kubernetes.io/projected/90ad213b-715b-45e7-9c75-77c7b0d9929a-kube-api-access-rqv2z\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:06.731892 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.731893    4528 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/90ad213b-715b-45e7-9c75-77c7b0d9929a-util\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 05:59:06.740658 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:06.740634    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"]
Nov 03 05:59:06.742235 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:06.742209    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680c9d72_4f8d_49c0_95cd_d25aab3710ef.slice/crio-b041abaa0edf2f168163a49835b8665501bfa38b10cb30d78e3e99767035481d WatchSource:0}: Error finding container b041abaa0edf2f168163a49835b8665501bfa38b10cb30d78e3e99767035481d: Status 404 returned error can't find the container with id b041abaa0edf2f168163a49835b8665501bfa38b10cb30d78e3e99767035481d
Nov 03 05:59:07.398935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:07.393813    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq" event={"ID":"90ad213b-715b-45e7-9c75-77c7b0d9929a","Type":"ContainerDied","Data":"750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d"}
Nov 03 05:59:07.398935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:07.393857    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7w5lbq"
Nov 03 05:59:07.398935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:07.393874    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="750a11a9d2211e8a81fa120ca2aaf390ff0862f7f4ed49a08d3962d2da21e81d"
Nov 03 05:59:07.398935 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:07.395732    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f" event={"ID":"680c9d72-4f8d-49c0-95cd-d25aab3710ef","Type":"ContainerStarted","Data":"b041abaa0edf2f168163a49835b8665501bfa38b10cb30d78e3e99767035481d"}
Nov 03 05:59:10.417118 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:10.417076    4528 generic.go:356] "Generic (PLEG): container finished" podID="680c9d72-4f8d-49c0-95cd-d25aab3710ef" containerID="a549904b6fff0958b9f47b2940a2a4fa73cdeefd9d144aabf088d9c49522dde6" exitCode=0
Nov 03 05:59:10.417118 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:10.417120    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f" event={"ID":"680c9d72-4f8d-49c0-95cd-d25aab3710ef","Type":"ContainerDied","Data":"a549904b6fff0958b9f47b2940a2a4fa73cdeefd9d144aabf088d9c49522dde6"}
Nov 03 05:59:11.423582 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:11.423547    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f" event={"ID":"680c9d72-4f8d-49c0-95cd-d25aab3710ef","Type":"ContainerStarted","Data":"32da78147161bd7d648210e9a19fce9ee0392147708f0bda404215b087a631f0"}
Nov 03 05:59:11.424010 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:11.423687    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:11.520636 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:11.520578    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f" podStartSLOduration=1.840074553 podStartE2EDuration="5.520560226s" totalImagesPullingTime="3.103209264s" totalInitContainerRuntime="577.276409ms" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:11.507509686 +0000 UTC m=+239.918906764" watchObservedRunningTime="2026-11-03 05:59:11.520560226 +0000 UTC m=+239.931957305"
Nov 03 05:59:13.528555 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528513    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"]
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528875    4528 cpu_manager.go:414] "containerMap: removing container" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="pull"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528890    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="pull"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528909    4528 cpu_manager.go:414] "containerMap: removing container" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="util"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528914    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="util"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528932    4528 cpu_manager.go:414] "containerMap: removing container" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="extract"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528938    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="extract"
Nov 03 05:59:13.529036 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.528986    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ad213b-715b-45e7-9c75-77c7b0d9929a" containerName="extract"
Nov 03 05:59:13.531725 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.531708    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.543597 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.543562    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"]
Nov 03 05:59:13.553181 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.553153    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:59:13.553181 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.553172    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-hk5j6\""
Nov 03 05:59:13.553386 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.553225    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Nov 03 05:59:13.691712 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:59:13.691668    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:59:13.693128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.693104    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.693249 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.693147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4vn7\" (UniqueName: \"kubernetes.io/projected/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-kube-api-access-n4vn7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.794484 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.794384    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.794484 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.794435    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4vn7\" (UniqueName: \"kubernetes.io/projected/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-kube-api-access-n4vn7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.794773 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.794752    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-tmp\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.817604 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.817576    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4vn7\" (UniqueName: \"kubernetes.io/projected/ce1f49bc-2dc7-4961-b182-886a69eb1ff1-kube-api-access-n4vn7\") pod \"cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq\" (UID: \"ce1f49bc-2dc7-4961-b182-886a69eb1ff1\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:13.843415 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:13.843381    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"
Nov 03 05:59:14.168579 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:14.168552    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq"]
Nov 03 05:59:14.170719 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:14.170687    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce1f49bc_2dc7_4961_b182_886a69eb1ff1.slice/crio-4313cb0f0e11c0d4acfe866f85ea534a089ea11763e60cce601ba097f3f9c47c WatchSource:0}: Error finding container 4313cb0f0e11c0d4acfe866f85ea534a089ea11763e60cce601ba097f3f9c47c: Status 404 returned error can't find the container with id 4313cb0f0e11c0d4acfe866f85ea534a089ea11763e60cce601ba097f3f9c47c
Nov 03 05:59:14.440098 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:14.440006    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq" event={"ID":"ce1f49bc-2dc7-4961-b182-886a69eb1ff1","Type":"ContainerStarted","Data":"4313cb0f0e11c0d4acfe866f85ea534a089ea11763e60cce601ba097f3f9c47c"}
Nov 03 05:59:16.452550 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:16.452508    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq" event={"ID":"ce1f49bc-2dc7-4961-b182-886a69eb1ff1","Type":"ContainerStarted","Data":"8c29513578148d44c2a7bb3533d13ebbbfb22777385bf361845b8ab80521287a"}
Nov 03 05:59:16.505913 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:16.505853    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-7d87bcd5fd-xd4pq" podStartSLOduration=2.129823563 podStartE2EDuration="3.505838735s" totalImagesPullingTime="1.376015172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:16.503393723 +0000 UTC m=+244.914790801" watchObservedRunningTime="2026-11-03 05:59:16.505838735 +0000 UTC m=+244.917235812"
Nov 03 05:59:18.881079 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.881041    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"]
Nov 03 05:59:18.883640 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.883616    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:18.915014 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.914976    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 03 05:59:18.915183 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.915038    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 03 05:59:18.925968 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.925937    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"]
Nov 03 05:59:18.932196 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:18.932173    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-rj97v\""
Nov 03 05:59:19.039624 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.039574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb86q\" (UniqueName: \"kubernetes.io/projected/5b6daa80-5641-4c74-b825-6830908c8de5-kube-api-access-zb86q\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.039784 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.039696    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b6daa80-5641-4c74-b825-6830908c8de5-tmp\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.140462 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.140364    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b6daa80-5641-4c74-b825-6830908c8de5-tmp\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.140630 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.140533    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb86q\" (UniqueName: \"kubernetes.io/projected/5b6daa80-5641-4c74-b825-6830908c8de5-kube-api-access-zb86q\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.140778 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.140759    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5b6daa80-5641-4c74-b825-6830908c8de5-tmp\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.163370 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.163333    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb86q\" (UniqueName: \"kubernetes.io/projected/5b6daa80-5641-4c74-b825-6830908c8de5-kube-api-access-zb86q\") pod \"jobset-operator-674cd59dc5-7kpgg\" (UID: \"5b6daa80-5641-4c74-b825-6830908c8de5\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.201290 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.201253    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"
Nov 03 05:59:19.507396 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:19.507309    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg"]
Nov 03 05:59:19.510389 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:19.510360    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b6daa80_5641_4c74_b825_6830908c8de5.slice/crio-71c46d47e5a9ff1eebd3404ed46186717e2c03f12af7b1005d383da406dcea5c WatchSource:0}: Error finding container 71c46d47e5a9ff1eebd3404ed46186717e2c03f12af7b1005d383da406dcea5c: Status 404 returned error can't find the container with id 71c46d47e5a9ff1eebd3404ed46186717e2c03f12af7b1005d383da406dcea5c
Nov 03 05:59:20.474450 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:20.474410    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg" event={"ID":"5b6daa80-5641-4c74-b825-6830908c8de5","Type":"ContainerStarted","Data":"71c46d47e5a9ff1eebd3404ed46186717e2c03f12af7b1005d383da406dcea5c"}
Nov 03 05:59:22.431568 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:22.431537    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f"
Nov 03 05:59:22.486560 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:22.486525    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg" event={"ID":"5b6daa80-5641-4c74-b825-6830908c8de5","Type":"ContainerStarted","Data":"9a713910acec352cbd3fd4a25d0542557f1f1ee396cb1e7c0c364de6d045e31a"}
Nov 03 05:59:22.564213 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:22.564160    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-7kpgg" podStartSLOduration=2.322022361 podStartE2EDuration="4.564143609s" totalImagesPullingTime="2.242121248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:22.562699266 +0000 UTC m=+250.974096344" watchObservedRunningTime="2026-11-03 05:59:22.564143609 +0000 UTC m=+250.975540687"
Nov 03 05:59:27.934031 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.933991    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cqcl"]
Nov 03 05:59:27.964461 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.964427    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cqcl"]
Nov 03 05:59:27.964637 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.964515    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:27.990310 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.990284    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-d68mz\""
Nov 03 05:59:27.990561 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.990540    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 05:59:27.990642 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:27.990565    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 05:59:28.032283 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.032258    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-bound-sa-token\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.032386 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.032294    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7d4\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-kube-api-access-lf7d4\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.132850 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.132822    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-bound-sa-token\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.132982 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.132868    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7d4\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-kube-api-access-lf7d4\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.159586 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.159559    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-bound-sa-token\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.159750 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.159730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7d4\" (UniqueName: \"kubernetes.io/projected/37ea6e44-7047-4b77-941d-7d08219b28f0-kube-api-access-lf7d4\") pod \"cert-manager-597b95cb84-5cqcl\" (UID: \"37ea6e44-7047-4b77-941d-7d08219b28f0\") " pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.173017 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.172993    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"]
Nov 03 05:59:28.193094 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.193037    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:28.199366 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.199337    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"]
Nov 03 05:59:28.209434 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.209411    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Nov 03 05:59:28.209434 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.209415    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bkgmr\""
Nov 03 05:59:28.212043 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.212021    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 05:59:28.234131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.234096    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cz7n\" (UniqueName: \"kubernetes.io/projected/491fbb46-3ea5-459b-be62-ab56797e6972-kube-api-access-2cz7n\") pod \"dns-operator-controller-manager-8557cbdb84-cmx7d\" (UID: \"491fbb46-3ea5-459b-be62-ab56797e6972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:28.295457 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.295426    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-5cqcl"
Nov 03 05:59:28.312836 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.312788    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 05:59:28.335363 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.335325    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cz7n\" (UniqueName: \"kubernetes.io/projected/491fbb46-3ea5-459b-be62-ab56797e6972-kube-api-access-2cz7n\") pod \"dns-operator-controller-manager-8557cbdb84-cmx7d\" (UID: \"491fbb46-3ea5-459b-be62-ab56797e6972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:28.381907 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.378317    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cz7n\" (UniqueName: \"kubernetes.io/projected/491fbb46-3ea5-459b-be62-ab56797e6972-kube-api-access-2cz7n\") pod \"dns-operator-controller-manager-8557cbdb84-cmx7d\" (UID: \"491fbb46-3ea5-459b-be62-ab56797e6972\") " pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:28.526162 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.526073    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:28.629500 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.629467    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-5cqcl"]
Nov 03 05:59:28.630786 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:28.630756    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37ea6e44_7047_4b77_941d_7d08219b28f0.slice/crio-6c4f75732f7a7b99b5d5211a611c41e93f38ca810bffb8621c57866245ff5704 WatchSource:0}: Error finding container 6c4f75732f7a7b99b5d5211a611c41e93f38ca810bffb8621c57866245ff5704: Status 404 returned error can't find the container with id 6c4f75732f7a7b99b5d5211a611c41e93f38ca810bffb8621c57866245ff5704
Nov 03 05:59:28.840872 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:28.840849    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"]
Nov 03 05:59:28.842511 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:28.842490    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod491fbb46_3ea5_459b_be62_ab56797e6972.slice/crio-eed29eaa7b884e8fdcbd0a1e91eed943237d94f4615a2e290099ca6176e50d8f WatchSource:0}: Error finding container eed29eaa7b884e8fdcbd0a1e91eed943237d94f4615a2e290099ca6176e50d8f: Status 404 returned error can't find the container with id eed29eaa7b884e8fdcbd0a1e91eed943237d94f4615a2e290099ca6176e50d8f
Nov 03 05:59:29.522675 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:29.522639    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d" event={"ID":"491fbb46-3ea5-459b-be62-ab56797e6972","Type":"ContainerStarted","Data":"eed29eaa7b884e8fdcbd0a1e91eed943237d94f4615a2e290099ca6176e50d8f"}
Nov 03 05:59:29.524313 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:29.524282    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5cqcl" event={"ID":"37ea6e44-7047-4b77-941d-7d08219b28f0","Type":"ContainerStarted","Data":"6c4f75732f7a7b99b5d5211a611c41e93f38ca810bffb8621c57866245ff5704"}
Nov 03 05:59:31.542143 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:31.542106    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-5cqcl" event={"ID":"37ea6e44-7047-4b77-941d-7d08219b28f0","Type":"ContainerStarted","Data":"1bc8123f9fdd2a223485249cadd6498a7424cdc5e8d642764916cc8d93484022"}
Nov 03 05:59:31.583742 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:31.583679    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-5cqcl" podStartSLOduration=1.792222886 podStartE2EDuration="4.583659869s" totalImagesPullingTime="2.791436983s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:31.581858428 +0000 UTC m=+259.993255507" watchObservedRunningTime="2026-11-03 05:59:31.583659869 +0000 UTC m=+259.995056946"
Nov 03 05:59:32.549771 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:32.549685    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d" event={"ID":"491fbb46-3ea5-459b-be62-ab56797e6972","Type":"ContainerStarted","Data":"5cf919169028f3c7bc18a63985ad1c7883e77a2d2b3072f8bef177d09b053d12"}
Nov 03 05:59:32.550265 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:32.549779    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:32.591591 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:32.591535    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d" podStartSLOduration=1.788780637 podStartE2EDuration="4.591519812s" totalImagesPullingTime="2.802739175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:32.590044178 +0000 UTC m=+261.001441262" watchObservedRunningTime="2026-11-03 05:59:32.591519812 +0000 UTC m=+261.002916889"
Nov 03 05:59:35.055788 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.055752    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 05:59:35.058942 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.058925    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.075654 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.075617    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 05:59:35.085385 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.085353    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lw66s\""
Nov 03 05:59:35.097712 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.097680    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.097846 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.097739    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8d4r\" (UniqueName: \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.198486 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.198442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8d4r\" (UniqueName: \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.198691 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.198542    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.198986 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.198962    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.237982 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.237941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8d4r\" (UniqueName: \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\") pod \"kuadrant-operator-controller-manager-cf8469d7b-5rjj2\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.371086 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.370999    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:35.711518 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:35.711493    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 05:59:36.573134 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:36.573093    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" event={"ID":"5bb92feb-d281-4404-92e0-e86e74a42a0c","Type":"ContainerStarted","Data":"562e3de2fc3df201d6b83e79b1fc2b057f46aa5a432b3e6ec6c706738644efef"}
Nov 03 05:59:38.676959 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.676918    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"]
Nov 03 05:59:38.679832 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.679812    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:38.710634 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.710605    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-n45lt\""
Nov 03 05:59:38.714131 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.714105    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"]
Nov 03 05:59:38.733180 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.733149    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqhb\" (UniqueName: \"kubernetes.io/projected/40c16b20-952a-40a3-892b-06a6dbfd0bb5-kube-api-access-7qqhb\") pod \"authorino-operator-7b7bb4bc77-xjzbk\" (UID: \"40c16b20-952a-40a3-892b-06a6dbfd0bb5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:38.834416 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.834367    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqhb\" (UniqueName: \"kubernetes.io/projected/40c16b20-952a-40a3-892b-06a6dbfd0bb5-kube-api-access-7qqhb\") pod \"authorino-operator-7b7bb4bc77-xjzbk\" (UID: \"40c16b20-952a-40a3-892b-06a6dbfd0bb5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:38.859303 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.859271    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqhb\" (UniqueName: \"kubernetes.io/projected/40c16b20-952a-40a3-892b-06a6dbfd0bb5-kube-api-access-7qqhb\") pod \"authorino-operator-7b7bb4bc77-xjzbk\" (UID: \"40c16b20-952a-40a3-892b-06a6dbfd0bb5\") " pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:38.994920 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:38.994839    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:39.296884 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:39.296788    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"]
Nov 03 05:59:39.299142 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:39.299111    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40c16b20_952a_40a3_892b_06a6dbfd0bb5.slice/crio-4456ec6477eba7066d6b3368fee3179944caf62272eb7abf158095e608e52937 WatchSource:0}: Error finding container 4456ec6477eba7066d6b3368fee3179944caf62272eb7abf158095e608e52937: Status 404 returned error can't find the container with id 4456ec6477eba7066d6b3368fee3179944caf62272eb7abf158095e608e52937
Nov 03 05:59:39.593171 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:39.593130    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk" event={"ID":"40c16b20-952a-40a3-892b-06a6dbfd0bb5","Type":"ContainerStarted","Data":"4456ec6477eba7066d6b3368fee3179944caf62272eb7abf158095e608e52937"}
Nov 03 05:59:40.602215 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.602181    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" event={"ID":"5bb92feb-d281-4404-92e0-e86e74a42a0c","Type":"ContainerStarted","Data":"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"}
Nov 03 05:59:40.602653 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.602329    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:40.690297 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.690237    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" podStartSLOduration=1.49648498 podStartE2EDuration="5.690218444s" totalImagesPullingTime="4.193733464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:40.688241962 +0000 UTC m=+269.099639040" watchObservedRunningTime="2026-11-03 05:59:40.690218444 +0000 UTC m=+269.101615522"
Nov 03 05:59:40.720841 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.720805    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"]
Nov 03 05:59:40.725694 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.725666    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.749226 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.749171    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"]
Nov 03 05:59:40.781786 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.781754    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-knqq5\""
Nov 03 05:59:40.856128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856035    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba44b32-7a71-462f-a683-7d6e39f140ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856362 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856252    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856362 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856317    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856463 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856437    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856522 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856567 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856539    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856638 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zscjc\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-kube-api-access-zscjc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.856745 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.856728    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957277 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957331    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zscjc\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-kube-api-access-zscjc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957388    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957421    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957459 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957450    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba44b32-7a71-462f-a683-7d6e39f140ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957649 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957739 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957713    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.957952 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.957932    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.958882 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.958181    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-data\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.958882 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.958549    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.958882 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.958828    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba44b32-7a71-462f-a683-7d6e39f140ac-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.960091 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.960062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.962233 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.962204    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.985307 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.985277    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zscjc\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-kube-api-access-zscjc\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:40.988357 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:40.988258    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba44b32-7a71-462f-a683-7d6e39f140ac-istio-token\") pod \"data-science-gateway-data-science-gateway-class-58c98c4485wjs6f\" (UID: \"aba44b32-7a71-462f-a683-7d6e39f140ac\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:41.042535 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.042499    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:41.378818 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.378758    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"]
Nov 03 05:59:41.380541 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:41.380505    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba44b32_7a71_462f_a683_7d6e39f140ac.slice/crio-b0e5164b69583c8007003c14c8144a16f0eab9f29c6840ebc41ceaab6233d2d8 WatchSource:0}: Error finding container b0e5164b69583c8007003c14c8144a16f0eab9f29c6840ebc41ceaab6233d2d8: Status 404 returned error can't find the container with id b0e5164b69583c8007003c14c8144a16f0eab9f29c6840ebc41ceaab6233d2d8
Nov 03 05:59:41.383281 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.383241    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:59:41.383362 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.383330    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:59:41.383399 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.383373    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 05:59:41.609238 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.609194    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk" event={"ID":"40c16b20-952a-40a3-892b-06a6dbfd0bb5","Type":"ContainerStarted","Data":"d7cba0a7c0f1d5d020b95b74b9a41d265a0dc46369e44fc188b4e7d5b2f285e0"}
Nov 03 05:59:41.609771 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.609748    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:41.611138 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.611111    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f" event={"ID":"aba44b32-7a71-462f-a683-7d6e39f140ac","Type":"ContainerStarted","Data":"be57eb790058d902e8c5eabf61a9617efdc7ebe2836baf624d82d3f0cf179741"}
Nov 03 05:59:41.611331 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.611167    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f" event={"ID":"aba44b32-7a71-462f-a683-7d6e39f140ac","Type":"ContainerStarted","Data":"b0e5164b69583c8007003c14c8144a16f0eab9f29c6840ebc41ceaab6233d2d8"}
Nov 03 05:59:41.654771 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.654657    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk" podStartSLOduration=1.8131085169999999 podStartE2EDuration="3.65464338s" totalImagesPullingTime="1.841534863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:41.652263771 +0000 UTC m=+270.063660842" watchObservedRunningTime="2026-11-03 05:59:41.65464338 +0000 UTC m=+270.066040456"
Nov 03 05:59:41.698276 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:41.698225    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f" podStartSLOduration=1.698210692 podStartE2EDuration="1.698210692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:41.697541892 +0000 UTC m=+270.108938991" watchObservedRunningTime="2026-11-03 05:59:41.698210692 +0000 UTC m=+270.109607768"
Nov 03 05:59:42.043385 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:42.043288    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:42.047929 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:42.047905    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:42.616986 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:42.616952    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:42.617970 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:42.617948    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-58c98c4485wjs6f"
Nov 03 05:59:43.558213 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:43.558184    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-8557cbdb84-cmx7d"
Nov 03 05:59:43.692261 ip-10-0-132-233 kubenswrapper[4528]: E1103 05:59:43.692228    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 05:59:44.365733 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.365693    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"]
Nov 03 05:59:44.374206 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.374177    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.387976 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.387949    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"]
Nov 03 05:59:44.397536 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.397500    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 03 05:59:44.400674 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.400650    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-qtbtg\""
Nov 03 05:59:44.400949 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.400929    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 03 05:59:44.494944 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.494910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7074acb-4264-4990-b61e-f30b57821f6d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.494944 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.494948    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28bk2\" (UniqueName: \"kubernetes.io/projected/b7074acb-4264-4990-b61e-f30b57821f6d-kube-api-access-28bk2\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.495143 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.494993    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7074acb-4264-4990-b61e-f30b57821f6d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.596205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.596134    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7074acb-4264-4990-b61e-f30b57821f6d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.596205 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.596209    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28bk2\" (UniqueName: \"kubernetes.io/projected/b7074acb-4264-4990-b61e-f30b57821f6d-kube-api-access-28bk2\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.596487 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.596255    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7074acb-4264-4990-b61e-f30b57821f6d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.596825 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.596779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/b7074acb-4264-4990-b61e-f30b57821f6d-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.598666 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.598644    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7074acb-4264-4990-b61e-f30b57821f6d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.622133 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.622074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28bk2\" (UniqueName: \"kubernetes.io/projected/b7074acb-4264-4990-b61e-f30b57821f6d-kube-api-access-28bk2\") pod \"kuadrant-console-plugin-66c4fb575d-vb997\" (UID: \"b7074acb-4264-4990-b61e-f30b57821f6d\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.687188 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.687151    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"
Nov 03 05:59:44.987183 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:44.987156    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997"]
Nov 03 05:59:44.989275 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:44.989238    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7074acb_4264_4990_b61e_f30b57821f6d.slice/crio-c12898f127fc1a3838c40a40ce04d27cb7829a861d5bcbd9c1e904210952a755 WatchSource:0}: Error finding container c12898f127fc1a3838c40a40ce04d27cb7829a861d5bcbd9c1e904210952a755: Status 404 returned error can't find the container with id c12898f127fc1a3838c40a40ce04d27cb7829a861d5bcbd9c1e904210952a755
Nov 03 05:59:45.635075 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:45.635036    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997" event={"ID":"b7074acb-4264-4990-b61e-f30b57821f6d","Type":"ContainerStarted","Data":"c12898f127fc1a3838c40a40ce04d27cb7829a861d5bcbd9c1e904210952a755"}
Nov 03 05:59:50.668048 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:50.668007    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997" event={"ID":"b7074acb-4264-4990-b61e-f30b57821f6d","Type":"ContainerStarted","Data":"8646295bbedb1f6eafe233b3490d8d8014ee53c2eb6b17614825a2b2030341d5"}
Nov 03 05:59:50.710008 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:50.709954    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-vb997" podStartSLOduration=1.7326205209999999 podStartE2EDuration="6.709937245s" totalImagesPullingTime="4.977316724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:50.709159759 +0000 UTC m=+279.120556837" watchObservedRunningTime="2026-11-03 05:59:50.709937245 +0000 UTC m=+279.121334323"
Nov 03 05:59:51.446476 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.446440    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"]
Nov 03 05:59:51.453272 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.453239    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.479092 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.479062    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 03 05:59:51.479270 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.479097    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 03 05:59:51.479270 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.479166    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-rxhj6\""
Nov 03 05:59:51.479270 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.479262    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 03 05:59:51.481608 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.481581    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"]
Nov 03 05:59:51.569926 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.569893    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-cert\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.570100 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.569949    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrvt4\" (UniqueName: \"kubernetes.io/projected/c6e0c543-0071-4228-a24b-c0c6c504ea4d-kube-api-access-nrvt4\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.570100 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.570026    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-metrics-certs\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.570100 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.570066    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6e0c543-0071-4228-a24b-c0c6c504ea4d-manager-config\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.613531 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.613495    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 05:59:51.670706 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.670668    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrvt4\" (UniqueName: \"kubernetes.io/projected/c6e0c543-0071-4228-a24b-c0c6c504ea4d-kube-api-access-nrvt4\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.671203 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.670735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-metrics-certs\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.671203 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.670822    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6e0c543-0071-4228-a24b-c0c6c504ea4d-manager-config\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.671203 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.670944    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-cert\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.671673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.671638    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/c6e0c543-0071-4228-a24b-c0c6c504ea4d-manager-config\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.674975 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.674924    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-metrics-certs\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.677327 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.677300    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c6e0c543-0071-4228-a24b-c0c6c504ea4d-cert\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.695147 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.695112    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrvt4\" (UniqueName: \"kubernetes.io/projected/c6e0c543-0071-4228-a24b-c0c6c504ea4d-kube-api-access-nrvt4\") pod \"jobset-controller-manager-f8868cd4b-pc7rp\" (UID: \"c6e0c543-0071-4228-a24b-c0c6c504ea4d\") " pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.766274 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.766188    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:51.948176 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.948118    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 05:59:51.952927 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.952905    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:51.979145 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:51.979112    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 05:59:52.075128 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.075093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.075331 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.075147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5t5nl\" (UniqueName: \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.096002 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.095977    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"]
Nov 03 05:59:52.097321 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:52.097297    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6e0c543_0071_4228_a24b_c0c6c504ea4d.slice/crio-344190c54a1557511abeb65bce5a10181a30d36cb09c3b06aea5022e1ce3fbb6 WatchSource:0}: Error finding container 344190c54a1557511abeb65bce5a10181a30d36cb09c3b06aea5022e1ce3fbb6: Status 404 returned error can't find the container with id 344190c54a1557511abeb65bce5a10181a30d36cb09c3b06aea5022e1ce3fbb6
Nov 03 05:59:52.176037 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.175999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.176218 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.176046    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5t5nl\" (UniqueName: \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.176398 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.176375    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.199330 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.199299    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5t5nl\" (UniqueName: \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\") pod \"kuadrant-operator-controller-manager-678d87766f-t49wh\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.266391 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.266364    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.577991 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.577963    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 05:59:52.579714 ip-10-0-132-233 kubenswrapper[4528]: W1103 05:59:52.579689    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc109b7c3_aa29_436e_9e3d_6c3e2e8bcdb4.slice/crio-4c22213cffbda40c89383a9072d90c484881935485167153fc6b3c03117ca5e4 WatchSource:0}: Error finding container 4c22213cffbda40c89383a9072d90c484881935485167153fc6b3c03117ca5e4: Status 404 returned error can't find the container with id 4c22213cffbda40c89383a9072d90c484881935485167153fc6b3c03117ca5e4
Nov 03 05:59:52.688246 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.688207    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" event={"ID":"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4","Type":"ContainerStarted","Data":"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"}
Nov 03 05:59:52.688862 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.688267    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" event={"ID":"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4","Type":"ContainerStarted","Data":"4c22213cffbda40c89383a9072d90c484881935485167153fc6b3c03117ca5e4"}
Nov 03 05:59:52.688862 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.688339    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 05:59:52.689673 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.689641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp" event={"ID":"c6e0c543-0071-4228-a24b-c0c6c504ea4d","Type":"ContainerStarted","Data":"344190c54a1557511abeb65bce5a10181a30d36cb09c3b06aea5022e1ce3fbb6"}
Nov 03 05:59:52.734902 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:52.734844    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" podStartSLOduration=1.734826086 podStartE2EDuration="1.734826086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:52.733115848 +0000 UTC m=+281.144512951" watchObservedRunningTime="2026-11-03 05:59:52.734826086 +0000 UTC m=+281.146223166"
Nov 03 05:59:53.626948 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:53.626917    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-7b7bb4bc77-xjzbk"
Nov 03 05:59:55.708516 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:55.708480    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp" event={"ID":"c6e0c543-0071-4228-a24b-c0c6c504ea4d","Type":"ContainerStarted","Data":"1029438af19422915ca475d741b86d794ecaace2a04a88061fa853caba9811a9"}
Nov 03 05:59:55.709023 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:55.708557    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 05:59:55.750324 ip-10-0-132-233 kubenswrapper[4528]: I1103 05:59:55.750271    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp" podStartSLOduration=2.086400839 podStartE2EDuration="4.750258078s" totalImagesPullingTime="2.663857239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 05:59:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 05:59:55.748498483 +0000 UTC m=+284.159895562" watchObservedRunningTime="2026-11-03 05:59:55.750258078 +0000 UTC m=+284.161655214"
Nov 03 06:00:03.699880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:03.699843    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 06:00:03.788705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:03.788671    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 06:00:03.788953 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:03.788923    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" podUID="5bb92feb-d281-4404-92e0-e86e74a42a0c" containerName="manager" containerID="cri-o://a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d" gracePeriod=10
Nov 03 06:00:04.043844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.043815    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 06:00:04.191593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.191554    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\") pod \"5bb92feb-d281-4404-92e0-e86e74a42a0c\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") "
Nov 03 06:00:04.191765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.191637    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\" (UniqueName: \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\") pod \"5bb92feb-d281-4404-92e0-e86e74a42a0c\" (UID: \"5bb92feb-d281-4404-92e0-e86e74a42a0c\") "
Nov 03 06:00:04.192229 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.192187    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume" pod "5bb92feb-d281-4404-92e0-e86e74a42a0c" (UID: "5bb92feb-d281-4404-92e0-e86e74a42a0c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:00:04.194024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.193996    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r" pod "5bb92feb-d281-4404-92e0-e86e74a42a0c" (UID: "5bb92feb-d281-4404-92e0-e86e74a42a0c"). InnerVolumeSpecName "kube-api-access-k8d4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:00:04.292614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.292526    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8d4r\" (UniqueName: \"kubernetes.io/projected/5bb92feb-d281-4404-92e0-e86e74a42a0c-kube-api-access-k8d4r\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:00:04.292614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.292559    4528 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5bb92feb-d281-4404-92e0-e86e74a42a0c-extensions-socket-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:00:04.755153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.755117    4528 generic.go:356] "Generic (PLEG): container finished" podID="5bb92feb-d281-4404-92e0-e86e74a42a0c" containerID="a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d" exitCode=0
Nov 03 06:00:04.755561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.755180    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"
Nov 03 06:00:04.755561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.755204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" event={"ID":"5bb92feb-d281-4404-92e0-e86e74a42a0c","Type":"ContainerDied","Data":"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"}
Nov 03 06:00:04.755561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.755257    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2" event={"ID":"5bb92feb-d281-4404-92e0-e86e74a42a0c","Type":"ContainerDied","Data":"562e3de2fc3df201d6b83e79b1fc2b057f46aa5a432b3e6ec6c706738644efef"}
Nov 03 06:00:04.755561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.755286    4528 scope.go:122] "RemoveContainer" containerID="a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"
Nov 03 06:00:04.765016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.764996    4528 scope.go:122] "RemoveContainer" containerID="a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"
Nov 03 06:00:04.765331 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:00:04.765306    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d\": container with ID starting with a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d not found: ID does not exist" containerID="a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"
Nov 03 06:00:04.765444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.765339    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d"} err="failed to get container status \"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d\": rpc error: code = NotFound desc = could not find container \"a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d\": container with ID starting with a2ee14b2cb1bf9378737b384e60c733cefdb81bf560ed7bc31f12fd31081415d not found: ID does not exist"
Nov 03 06:00:04.834426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.834343    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 06:00:04.845776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:04.845746    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-cf8469d7b-5rjj2"]
Nov 03 06:00:05.746544 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:05.746500    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb92feb-d281-4404-92e0-e86e74a42a0c" path="/var/lib/kubelet/pods/5bb92feb-d281-4404-92e0-e86e74a42a0c/volumes"
Nov 03 06:00:06.719096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:00:06.719069    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp"
Nov 03 06:00:13.692036 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:00:13.691992    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:00:43.698014 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:00:43.697923    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:01:13.692888 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:01:13.692834    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:01:43.712998 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:01:43.712957    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:01:51.349609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.349573    4528 generic.go:356] "Generic (PLEG): container finished" podID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerID="f093fdbe432791d0aa5949197047ad4092d88808fe7cbdb3d654ea29d310cb38" exitCode=0
Nov 03 06:01:51.350071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.349643    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" event={"ID":"e8c4a3ba-5a15-40ac-823e-795c43744b3c","Type":"ContainerDied","Data":"f093fdbe432791d0aa5949197047ad4092d88808fe7cbdb3d654ea29d310cb38"}
Nov 03 06:01:51.350071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.350046    4528 scope.go:122] "RemoveContainer" containerID="f093fdbe432791d0aa5949197047ad4092d88808fe7cbdb3d654ea29d310cb38"
Nov 03 06:01:51.351163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.351134    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:01:51.351230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.351208    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:01:51.351265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:51.351242    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:01:52.357039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:52.357003    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" event={"ID":"e8c4a3ba-5a15-40ac-823e-795c43744b3c","Type":"ContainerStarted","Data":"d8abe28cfbcaff1373248fffe6fca95e57e349dfc8c43764c059d7cd4df258c9"}
Nov 03 06:01:52.612629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:52.612539    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 06:01:52.613665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:52.613637    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Nov 03 06:01:52.613769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:52.613690    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Nov 03 06:01:53.612537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:53.612498    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Nov 03 06:01:53.612959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:53.612557    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Nov 03 06:01:54.612758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:54.612719    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Nov 03 06:01:54.613177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:54.612776    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Nov 03 06:01:54.911111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:54.911001    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 06:01:55.613166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:55.613123    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Nov 03 06:01:55.613584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:55.613181    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Nov 03 06:01:56.613236 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:56.613192    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused" start-of-body=
Nov 03 06:01:56.613637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:56.613251    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": dial tcp 10.132.0.28:15021: connect: connection refused"
Nov 03 06:01:57.616703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:57.616674    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 06:01:57.617975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:01:57.617947    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk"
Nov 03 06:02:13.692906 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:02:13.692860    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:02:40.703499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.703446    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"]
Nov 03 06:02:40.704206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.703877    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5bb92feb-d281-4404-92e0-e86e74a42a0c" containerName="manager"
Nov 03 06:02:40.704206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.703895    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb92feb-d281-4404-92e0-e86e74a42a0c" containerName="manager"
Nov 03 06:02:40.704206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.704009    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb92feb-d281-4404-92e0-e86e74a42a0c" containerName="manager"
Nov 03 06:02:40.707113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.707090    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.724318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.724284    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 06:02:40.735441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.735414    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 06:02:40.735605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.735531    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 06:02:40.735715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.735704    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t4d9g\""
Nov 03 06:02:40.738217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.738194    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"]
Nov 03 06:02:40.778265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778227    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.778443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778298    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.778587 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778567    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzlsr\" (UniqueName: \"kubernetes.io/projected/15775b64-f27a-4c4a-8eeb-3455396727c9-kube-api-access-zzlsr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.778652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778631    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.778652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778646    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15775b64-f27a-4c4a-8eeb-3455396727c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.778818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.778775    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880101    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzlsr\" (UniqueName: \"kubernetes.io/projected/15775b64-f27a-4c4a-8eeb-3455396727c9-kube-api-access-zzlsr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880177    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880203    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15775b64-f27a-4c4a-8eeb-3455396727c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880283    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880328    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.880712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.882094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.880967    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.883639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.883560    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/15775b64-f27a-4c4a-8eeb-3455396727c9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.883870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.883776    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/15775b64-f27a-4c4a-8eeb-3455396727c9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:40.903040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:40.903010    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzlsr\" (UniqueName: \"kubernetes.io/projected/15775b64-f27a-4c4a-8eeb-3455396727c9-kube-api-access-zzlsr\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv\" (UID: \"15775b64-f27a-4c4a-8eeb-3455396727c9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:41.020237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:41.020141    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:41.323894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:41.323852    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"]
Nov 03 06:02:41.327244 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:02:41.327209    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15775b64_f27a_4c4a_8eeb_3455396727c9.slice/crio-10d70402f6e3187ddd451aaa97e80bff8dfc454a11455a62ed558ee1ebd24227 WatchSource:0}: Error finding container 10d70402f6e3187ddd451aaa97e80bff8dfc454a11455a62ed558ee1ebd24227: Status 404 returned error can't find the container with id 10d70402f6e3187ddd451aaa97e80bff8dfc454a11455a62ed558ee1ebd24227
Nov 03 06:02:41.328429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:41.328409    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 06:02:41.633346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:41.633250    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv" event={"ID":"15775b64-f27a-4c4a-8eeb-3455396727c9","Type":"ContainerStarted","Data":"10d70402f6e3187ddd451aaa97e80bff8dfc454a11455a62ed558ee1ebd24227"}
Nov 03 06:02:42.640235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:42.640202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv" event={"ID":"15775b64-f27a-4c4a-8eeb-3455396727c9","Type":"ContainerStarted","Data":"24ed28ba3886a9943d43612c522eea6d0d62b933e6edf8e05c98433bee3b33fa"}
Nov 03 06:02:42.680885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:42.680823    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv" podStartSLOduration=1.490429358 podStartE2EDuration="2.680804991s" totalImagesPullingTime="1.190375633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:42.680348059 +0000 UTC m=+451.091745137" watchObservedRunningTime="2026-11-03 06:02:42.680804991 +0000 UTC m=+451.092202065"
Nov 03 06:02:43.020816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.020759    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:43.478128 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.478091    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"]
Nov 03 06:02:43.482153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.482126    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.499583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.499556    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 06:02:43.514883 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.514850    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"]
Nov 03 06:02:43.606186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606149    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.606365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606195    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.606365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606313    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.606365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606359    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.606539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.606539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.606494    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-742j9\" (UniqueName: \"kubernetes.io/projected/6a36b5bf-8092-44f9-9317-d1d1120fe40d-kube-api-access-742j9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.695588 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:02:43.695545    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:02:43.707498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707464    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.707498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707509    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.707755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707568    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.707755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707593    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-742j9\" (UniqueName: \"kubernetes.io/projected/6a36b5bf-8092-44f9-9317-d1d1120fe40d-kube-api-access-742j9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.707755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707644    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.707755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.707692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.708100 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.708074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.708458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.708416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.708688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.708667    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.710292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.710250    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a36b5bf-8092-44f9-9317-d1d1120fe40d-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.710292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.710267    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a36b5bf-8092-44f9-9317-d1d1120fe40d-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.734238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.734153    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-742j9\" (UniqueName: \"kubernetes.io/projected/6a36b5bf-8092-44f9-9317-d1d1120fe40d-kube-api-access-742j9\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-b8h79\" (UID: \"6a36b5bf-8092-44f9-9317-d1d1120fe40d\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:43.800021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:43.799983    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:44.109959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.109929    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"]
Nov 03 06:02:44.112408 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:02:44.112379    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a36b5bf_8092_44f9_9317_d1d1120fe40d.slice/crio-0a801c9d831ee63c547560b042d1b6bcce7fc0fcd1b99ebf6fc76e3e8cba94ce WatchSource:0}: Error finding container 0a801c9d831ee63c547560b042d1b6bcce7fc0fcd1b99ebf6fc76e3e8cba94ce: Status 404 returned error can't find the container with id 0a801c9d831ee63c547560b042d1b6bcce7fc0fcd1b99ebf6fc76e3e8cba94ce
Nov 03 06:02:44.657815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.657684    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79" event={"ID":"6a36b5bf-8092-44f9-9317-d1d1120fe40d","Type":"ContainerStarted","Data":"6dff564e8eec15902a932922d48e3a4f315c3b3188de73bdecb5727c692abf60"}
Nov 03 06:02:44.657993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.657823    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79" event={"ID":"6a36b5bf-8092-44f9-9317-d1d1120fe40d","Type":"ContainerStarted","Data":"0a801c9d831ee63c547560b042d1b6bcce7fc0fcd1b99ebf6fc76e3e8cba94ce"}
Nov 03 06:02:44.690785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.690748    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"]
Nov 03 06:02:44.694956 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.694932    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.711217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.711074    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79" podStartSLOduration=1.49494353 podStartE2EDuration="1.711054864s" totalImagesPullingTime="216.111334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:44.705910256 +0000 UTC m=+453.117307334" watchObservedRunningTime="2026-11-03 06:02:44.711054864 +0000 UTC m=+453.122451942"
Nov 03 06:02:44.714127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.713908    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"]
Nov 03 06:02:44.716828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.716788    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 03 06:02:44.800910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.800875    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:44.818353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818312    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.818515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818361    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.818515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818435    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sstzf\" (UniqueName: \"kubernetes.io/projected/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-kube-api-access-sstzf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.818515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818465    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.818515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818484    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.818702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.818681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.919917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.919823    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.919917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.919904    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.919949    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.919980    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.920011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sstzf\" (UniqueName: \"kubernetes.io/projected/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-kube-api-access-sstzf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.920037    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.920388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.920423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.920535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.920459    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.922440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.922410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.922521 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.922416    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:44.942721 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:44.942697    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sstzf\" (UniqueName: \"kubernetes.io/projected/3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5-kube-api-access-sstzf\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-dxpcj\" (UID: \"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:45.010516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:45.010471    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:45.312880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:45.312840    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"]
Nov 03 06:02:45.315740 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:02:45.315709    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e692fdf_8ed2_4ccc_8cb3_86f39898b3d5.slice/crio-82292ef63514cabe58fe77aa585b1acd0e55b6093549a5c7d75501d8104f0e38 WatchSource:0}: Error finding container 82292ef63514cabe58fe77aa585b1acd0e55b6093549a5c7d75501d8104f0e38: Status 404 returned error can't find the container with id 82292ef63514cabe58fe77aa585b1acd0e55b6093549a5c7d75501d8104f0e38
Nov 03 06:02:45.664353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:45.664311    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj" event={"ID":"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5","Type":"ContainerStarted","Data":"72e33bea5bd8079f09a1820b515336aa99c111f98c0c33493b26db50a5f3fc49"}
Nov 03 06:02:45.664566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:45.664368    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj" event={"ID":"3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5","Type":"ContainerStarted","Data":"82292ef63514cabe58fe77aa585b1acd0e55b6093549a5c7d75501d8104f0e38"}
Nov 03 06:02:45.710229 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:45.710168    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj" podStartSLOduration=1.499630524 podStartE2EDuration="1.710152731s" totalImagesPullingTime="210.522207ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:45.706099214 +0000 UTC m=+454.117496294" watchObservedRunningTime="2026-11-03 06:02:45.710152731 +0000 UTC m=+454.121549835"
Nov 03 06:02:46.011698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:46.011608    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:50.843556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.843513    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"]
Nov 03 06:02:50.847465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.847442    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.872706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.872667    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"]
Nov 03 06:02:50.873730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.873708    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 06:02:50.984416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984368    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.984614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984452    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks9gm\" (UniqueName: \"kubernetes.io/projected/68da6dd2-14bd-413e-ab3b-0118d96be1b0-kube-api-access-ks9gm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.984614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984576    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.984737 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984630    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.984788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984764    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:50.985039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:50.984843    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.021343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.021307    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:51.035943 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.035905    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:51.085757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.085699    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.085961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.085812    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.085961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.085845    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.085961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.085957    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.086070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.086015    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks9gm\" (UniqueName: \"kubernetes.io/projected/68da6dd2-14bd-413e-ab3b-0118d96be1b0-kube-api-access-ks9gm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.086070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.086057    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.086237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.086213    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.086441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.086421    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.086531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.086510    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.089086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.089055    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68da6dd2-14bd-413e-ab3b-0118d96be1b0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.089625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.089604    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68da6dd2-14bd-413e-ab3b-0118d96be1b0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.110567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.110490    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks9gm\" (UniqueName: \"kubernetes.io/projected/68da6dd2-14bd-413e-ab3b-0118d96be1b0-kube-api-access-ks9gm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-4btct\" (UID: \"68da6dd2-14bd-413e-ab3b-0118d96be1b0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.162767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.162729    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:51.470351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.470321    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"]
Nov 03 06:02:51.473085 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:02:51.473056    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68da6dd2_14bd_413e_ab3b_0118d96be1b0.slice/crio-c1fa497e7405c386c1a24b027cf84c672537a8d2df39caacf197893b68d998b9 WatchSource:0}: Error finding container c1fa497e7405c386c1a24b027cf84c672537a8d2df39caacf197893b68d998b9: Status 404 returned error can't find the container with id c1fa497e7405c386c1a24b027cf84c672537a8d2df39caacf197893b68d998b9
Nov 03 06:02:51.701055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.701022    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct" event={"ID":"68da6dd2-14bd-413e-ab3b-0118d96be1b0","Type":"ContainerStarted","Data":"c1fa497e7405c386c1a24b027cf84c672537a8d2df39caacf197893b68d998b9"}
Nov 03 06:02:51.712422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:51.712397    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv"
Nov 03 06:02:52.709388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:52.709350    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct" event={"ID":"68da6dd2-14bd-413e-ab3b-0118d96be1b0","Type":"ContainerStarted","Data":"cbb361c83e6ad65de90c72410054d51058618f4f52c23aba035f7039d81c33f1"}
Nov 03 06:02:52.751263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:52.751206    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct" podStartSLOduration=2.544308453 podStartE2EDuration="2.751193214s" totalImagesPullingTime="206.884761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:02:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:02:52.749600542 +0000 UTC m=+461.160997616" watchObservedRunningTime="2026-11-03 06:02:52.751193214 +0000 UTC m=+461.162590292"
Nov 03 06:02:53.163552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:53.163504    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:02:53.800504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:53.800469    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:53.813122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:53.813097    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:54.732942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:54.732912    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-b8h79"
Nov 03 06:02:55.011494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:55.011404    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:55.024401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:55.024367    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:02:55.747631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:02:55.747603    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-dxpcj"
Nov 03 06:03:01.163834 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:01.163780    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:03:01.176581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:01.176552    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:03:01.773354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:01.773328    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-4btct"
Nov 03 06:03:10.639296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.639253    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:10.642101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.642076    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.658409 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.658371    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 06:03:10.659019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.658994    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 06:03:10.659019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.658995    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 06:03:10.670257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.670223    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 06:03:10.670442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.670395    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-ssgfm\""
Nov 03 06:03:10.675372 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.675340    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:10.687821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.687759    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64v6\" (UniqueName: \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.688006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.687927    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.789054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.789012    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64v6\" (UniqueName: \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.789373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.789340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.789481 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:03:10.789422    4528 secret.go:189] Couldn't get secret opendatahub/ai-gateway-controller-metrics-tls: secret "ai-gateway-controller-metrics-tls" not found
Nov 03 06:03:10.789525 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:03:10.789486    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs podName:48cf3c21-d4ce-469e-ab77-25cc5b979dd1 nodeName:}" failed. No retries permitted until 2026-11-03 06:03:11.289471975 +0000 UTC m=+479.700869031 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs") pod "ai-gateway-controller-b59d66cb5-qbstw" (UID: "48cf3c21-d4ce-469e-ab77-25cc5b979dd1") : secret "ai-gateway-controller-metrics-tls" not found
Nov 03 06:03:10.812505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.812467    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64v6\" (UniqueName: \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:10.890061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.889974    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"]
Nov 03 06:03:10.892650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.892632    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.917055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.917020    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"]
Nov 03 06:03:10.990731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990695    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11852422-e9b2-4cc4-a315-644dd38888f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.990731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990734    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990836    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11852422-e9b2-4cc4-a315-644dd38888f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990876    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990892    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.990910    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.991039    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.991135    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:10.991212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:10.991185    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxd7x\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-kube-api-access-hxd7x\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.091892 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.091852    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.091892 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.091905    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxd7x\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-kube-api-access-hxd7x\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092243 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092222    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11852422-e9b2-4cc4-a315-644dd38888f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092299 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092258    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092313    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11852422-e9b2-4cc4-a315-644dd38888f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092438    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092629    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092668    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.092754 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.092720    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.093055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.093035    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/11852422-e9b2-4cc4-a315-644dd38888f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.094654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.094637    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/11852422-e9b2-4cc4-a315-644dd38888f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.094804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.094776    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/11852422-e9b2-4cc4-a315-644dd38888f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.115083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.115047    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.115224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.115176    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxd7x\" (UniqueName: \"kubernetes.io/projected/11852422-e9b2-4cc4-a315-644dd38888f9-kube-api-access-hxd7x\") pod \"maas-default-gateway-openshift-default-778b7b5c47-8mbmr\" (UID: \"11852422-e9b2-4cc4-a315-644dd38888f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.208266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.208174    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:11.294668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.294635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:11.297934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.297904    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") pod \"ai-gateway-controller-b59d66cb5-qbstw\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") " pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:11.523709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.523677    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"]
Nov 03 06:03:11.525313 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:03:11.525287    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11852422_e9b2_4cc4_a315_644dd38888f9.slice/crio-b2062ab3d7ced6ec6335cc56b6d998724c3f778b533a5576b4a34f61481d759a WatchSource:0}: Error finding container b2062ab3d7ced6ec6335cc56b6d998724c3f778b533a5576b4a34f61481d759a: Status 404 returned error can't find the container with id b2062ab3d7ced6ec6335cc56b6d998724c3f778b533a5576b4a34f61481d759a
Nov 03 06:03:11.528211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.528164    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:03:11.528333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.528246    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:03:11.528333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.528294    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:03:11.556669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.556640    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:11.826138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.826103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" event={"ID":"11852422-e9b2-4cc4-a315-644dd38888f9","Type":"ContainerStarted","Data":"c6e5a359110a67ed2a19530bd5043fccd22611bec77263eaba73e3ba4838f8eb"}
Nov 03 06:03:11.826699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.826170    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" event={"ID":"11852422-e9b2-4cc4-a315-644dd38888f9","Type":"ContainerStarted","Data":"b2062ab3d7ced6ec6335cc56b6d998724c3f778b533a5576b4a34f61481d759a"}
Nov 03 06:03:11.865165 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.865128    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:11.866125 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:03:11.866091    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48cf3c21_d4ce_469e_ab77_25cc5b979dd1.slice/crio-fec47be178b6247b55b77ff40b2cea4cbfb985862ca7902bd982a28aee71d5fb WatchSource:0}: Error finding container fec47be178b6247b55b77ff40b2cea4cbfb985862ca7902bd982a28aee71d5fb: Status 404 returned error can't find the container with id fec47be178b6247b55b77ff40b2cea4cbfb985862ca7902bd982a28aee71d5fb
Nov 03 06:03:11.870875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:11.870816    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podStartSLOduration=1.870782496 podStartE2EDuration="1.870782496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:11.869875207 +0000 UTC m=+480.281272281" watchObservedRunningTime="2026-11-03 06:03:11.870782496 +0000 UTC m=+480.282179575"
Nov 03 06:03:12.209189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:12.209098    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:12.210390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:12.210367    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:12.210447 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:12.210417    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:12.835717 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:12.835658    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" event={"ID":"48cf3c21-d4ce-469e-ab77-25cc5b979dd1","Type":"ContainerStarted","Data":"fec47be178b6247b55b77ff40b2cea4cbfb985862ca7902bd982a28aee71d5fb"}
Nov 03 06:03:13.212878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:13.211244    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:13.212878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:13.211336    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:13.731631 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:03:13.731546    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:03:14.209412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:14.209373    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:14.209977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:14.209427    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:15.067646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.067619    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 06:03:15.209606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.209565    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:15.210066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.209636    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:15.854816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.854741    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" event={"ID":"48cf3c21-d4ce-469e-ab77-25cc5b979dd1","Type":"ContainerStarted","Data":"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"}
Nov 03 06:03:15.855031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.854861    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:15.900922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:15.900855    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" podStartSLOduration=2.714959384 podStartE2EDuration="5.90083668s" totalImagesPullingTime="3.185877296s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:15.900206234 +0000 UTC m=+484.311603312" watchObservedRunningTime="2026-11-03 06:03:15.90083668 +0000 UTC m=+484.312233759"
Nov 03 06:03:16.209148 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:16.209041    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:16.209148 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:16.209106    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:17.209047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.208997    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:17.209462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.209076    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:17.531141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.531050    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fggx5"]
Nov 03 06:03:17.538664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.538629    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.553313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.553279    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fggx5"]
Nov 03 06:03:17.561046 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.561019    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-zlcl8\""
Nov 03 06:03:17.561214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.561052    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 06:03:17.567451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.567424    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.567616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.567596    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq4ff\" (UniqueName: \"kubernetes.io/projected/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-kube-api-access-jq4ff\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.595971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.595932    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"]
Nov 03 06:03:17.601014 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.600977    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.617336 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.617303    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"]
Nov 03 06:03:17.678821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.668551    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq4ff\" (UniqueName: \"kubernetes.io/projected/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-kube-api-access-jq4ff\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.678821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.668615    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.678821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.668648    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c767011-9f84-4a57-9d95-727814cc25e4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.678821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.668699    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4hxt\" (UniqueName: \"kubernetes.io/projected/1c767011-9f84-4a57-9d95-727814cc25e4-kube-api-access-m4hxt\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.678821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.670244    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.694748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.694702    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq4ff\" (UniqueName: \"kubernetes.io/projected/ddfcd2d1-6719-44f9-bff2-b0d5c937b50b-kube-api-access-jq4ff\") pod \"payload-processing-55fbbbf9dc-fggx5\" (UID: \"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.770211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.770172    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4hxt\" (UniqueName: \"kubernetes.io/projected/1c767011-9f84-4a57-9d95-727814cc25e4-kube-api-access-m4hxt\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.770432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.770336    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c767011-9f84-4a57-9d95-727814cc25e4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.771353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.771327    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c767011-9f84-4a57-9d95-727814cc25e4-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.794773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.794680    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4hxt\" (UniqueName: \"kubernetes.io/projected/1c767011-9f84-4a57-9d95-727814cc25e4-kube-api-access-m4hxt\") pod \"payload-pre-processing-7b98bb678c-9m2mq\" (UID: \"1c767011-9f84-4a57-9d95-727814cc25e4\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:17.856355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.856319    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:17.919473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:17.919443    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:18.164359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.164330    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-fggx5"]
Nov 03 06:03:18.165927 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:03:18.165902    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfcd2d1_6719_44f9_bff2_b0d5c937b50b.slice/crio-cc9c26a985cb1180a0bef765a6d589570658817df653d33c0dc738f4de111319 WatchSource:0}: Error finding container cc9c26a985cb1180a0bef765a6d589570658817df653d33c0dc738f4de111319: Status 404 returned error can't find the container with id cc9c26a985cb1180a0bef765a6d589570658817df653d33c0dc738f4de111319
Nov 03 06:03:18.209401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.209368    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:18.209996 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.209421    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:18.269747 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.269704    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"]
Nov 03 06:03:18.270336 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:03:18.270311    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c767011_9f84_4a57_9d95_727814cc25e4.slice/crio-db9ce4d189ca0e9687f4a08d326e71ecb177287c1d2f6763c57678eb1c42c988 WatchSource:0}: Error finding container db9ce4d189ca0e9687f4a08d326e71ecb177287c1d2f6763c57678eb1c42c988: Status 404 returned error can't find the container with id db9ce4d189ca0e9687f4a08d326e71ecb177287c1d2f6763c57678eb1c42c988
Nov 03 06:03:18.884591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.884493    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq" event={"ID":"1c767011-9f84-4a57-9d95-727814cc25e4","Type":"ContainerStarted","Data":"db9ce4d189ca0e9687f4a08d326e71ecb177287c1d2f6763c57678eb1c42c988"}
Nov 03 06:03:18.887155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:18.887062    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5" event={"ID":"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b","Type":"ContainerStarted","Data":"cc9c26a985cb1180a0bef765a6d589570658817df653d33c0dc738f4de111319"}
Nov 03 06:03:19.225342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:19.225254    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body=
Nov 03 06:03:19.225342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:19.225315    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused"
Nov 03 06:03:21.209984 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.209939    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:03:21.210416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.210016    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:03:21.916525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.916481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5" event={"ID":"ddfcd2d1-6719-44f9-bff2-b0d5c937b50b","Type":"ContainerStarted","Data":"d4c18f52e138fb2f6ec8398a1007def13dd5da3b4ef63dbf932f8ac31ea77956"}
Nov 03 06:03:21.916860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.916833    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:21.918822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.918785    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq" event={"ID":"1c767011-9f84-4a57-9d95-727814cc25e4","Type":"ContainerStarted","Data":"9df4d89ba3712e6b9106bb17cf529773644ae4ccf0427410572919b302d765aa"}
Nov 03 06:03:21.918950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.918919    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:21.958879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.958817    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5" podStartSLOduration=1.9157900159999999 podStartE2EDuration="4.958801728s" totalImagesPullingTime="3.043011712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:21.956572822 +0000 UTC m=+490.367969905" watchObservedRunningTime="2026-11-03 06:03:21.958801728 +0000 UTC m=+490.370198797"
Nov 03 06:03:21.996646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:21.996580    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq" podStartSLOduration=2.096267453 podStartE2EDuration="4.996561009s" totalImagesPullingTime="2.900293556s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:21.996132758 +0000 UTC m=+490.407529836" watchObservedRunningTime="2026-11-03 06:03:21.996561009 +0000 UTC m=+490.407958088"
Nov 03 06:03:22.252041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:22.251946    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:03:22.252041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:22.252014    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:03:22.734070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:22.734037    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:22.734508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:22.734463    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:22.735597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:22.735575    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr"
Nov 03 06:03:26.862932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:26.862849    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:27.135187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.135093    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"]
Nov 03 06:03:27.138664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.138642    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.157062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.157030    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"]
Nov 03 06:03:27.168929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.168894    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnz52\" (UniqueName: \"kubernetes.io/projected/0c7a3d80-23c6-435f-8847-98330817b665-kube-api-access-jnz52\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.169081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.169036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7a3d80-23c6-435f-8847-98330817b665-metrics-certs\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.270302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.270257    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnz52\" (UniqueName: \"kubernetes.io/projected/0c7a3d80-23c6-435f-8847-98330817b665-kube-api-access-jnz52\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.270498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.270348    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7a3d80-23c6-435f-8847-98330817b665-metrics-certs\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.272845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.272813    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c7a3d80-23c6-435f-8847-98330817b665-metrics-certs\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.293772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.293738    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnz52\" (UniqueName: \"kubernetes.io/projected/0c7a3d80-23c6-435f-8847-98330817b665-kube-api-access-jnz52\") pod \"ai-gateway-controller-7f7fb74d9c-c8g4b\" (UID: \"0c7a3d80-23c6-435f-8847-98330817b665\") " pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.452535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.452422    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:27.753844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.753813    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"]
Nov 03 06:03:27.755714 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:03:27.755676    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c7a3d80_23c6_435f_8847_98330817b665.slice/crio-dd0dba852f78ad7bcb80b4b9fade8b09362f3753dce7483fb42a3f53226a82b7 WatchSource:0}: Error finding container dd0dba852f78ad7bcb80b4b9fade8b09362f3753dce7483fb42a3f53226a82b7: Status 404 returned error can't find the container with id dd0dba852f78ad7bcb80b4b9fade8b09362f3753dce7483fb42a3f53226a82b7
Nov 03 06:03:27.953570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:27.953530    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b" event={"ID":"0c7a3d80-23c6-435f-8847-98330817b665","Type":"ContainerStarted","Data":"dd0dba852f78ad7bcb80b4b9fade8b09362f3753dce7483fb42a3f53226a82b7"}
Nov 03 06:03:28.961077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:28.961041    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b" event={"ID":"0c7a3d80-23c6-435f-8847-98330817b665","Type":"ContainerStarted","Data":"478867734eeaf63c48dd97b6bac074e0d78729d49a4c3bab4fceb69cbdca6c0f"}
Nov 03 06:03:28.961495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:28.961180    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:29.004342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:29.004284    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b" podStartSLOduration=1.6139527999999999 podStartE2EDuration="2.004269465s" totalImagesPullingTime="390.316665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:03:29.001386867 +0000 UTC m=+497.412783947" watchObservedRunningTime="2026-11-03 06:03:29.004269465 +0000 UTC m=+497.415666543"
Nov 03 06:03:32.927130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:32.927099    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-fggx5"
Nov 03 06:03:32.927536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:32.927167    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9m2mq"
Nov 03 06:03:39.970070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:39.970037    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7f7fb74d9c-c8g4b"
Nov 03 06:03:40.042532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.042486    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:40.042811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.042749    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" podUID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" containerName="manager" containerID="cri-o://899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa" gracePeriod=30
Nov 03 06:03:40.296640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.296608    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:40.397776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.397733    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\" (UniqueName: \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\") pod \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") "
Nov 03 06:03:40.397994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.397882    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") pod \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\" (UID: \"48cf3c21-d4ce-469e-ab77-25cc5b979dd1\") "
Nov 03 06:03:40.400260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.400218    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6" pod "48cf3c21-d4ce-469e-ab77-25cc5b979dd1" (UID: "48cf3c21-d4ce-469e-ab77-25cc5b979dd1"). InnerVolumeSpecName "kube-api-access-n64v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:03:40.400381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.400252    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs" pod "48cf3c21-d4ce-469e-ab77-25cc5b979dd1" (UID: "48cf3c21-d4ce-469e-ab77-25cc5b979dd1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:03:40.499399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.499362    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n64v6\" (UniqueName: \"kubernetes.io/projected/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-kube-api-access-n64v6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:03:40.499399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:40.499397    4528 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/48cf3c21-d4ce-469e-ab77-25cc5b979dd1-metrics-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:03:41.038101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.038064    4528 generic.go:356] "Generic (PLEG): container finished" podID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" containerID="899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa" exitCode=0
Nov 03 06:03:41.038101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.038106    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw"
Nov 03 06:03:41.038622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.038139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" event={"ID":"48cf3c21-d4ce-469e-ab77-25cc5b979dd1","Type":"ContainerDied","Data":"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"}
Nov 03 06:03:41.038622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.038182    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b59d66cb5-qbstw" event={"ID":"48cf3c21-d4ce-469e-ab77-25cc5b979dd1","Type":"ContainerDied","Data":"fec47be178b6247b55b77ff40b2cea4cbfb985862ca7902bd982a28aee71d5fb"}
Nov 03 06:03:41.038622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.038208    4528 scope.go:122] "RemoveContainer" containerID="899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"
Nov 03 06:03:41.048357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.048180    4528 scope.go:122] "RemoveContainer" containerID="899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"
Nov 03 06:03:41.048548 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:03:41.048527    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa\": container with ID starting with 899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa not found: ID does not exist" containerID="899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"
Nov 03 06:03:41.048591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.048559    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa"} err="failed to get container status \"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa\": rpc error: code = NotFound desc = could not find container \"899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa\": container with ID starting with 899f6eb0da6f6d9c0357aefc90f62d584c660fbfebaf65012d15eb40083024fa not found: ID does not exist"
Nov 03 06:03:41.088097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.088063    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:41.104818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.101743    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-b59d66cb5-qbstw"]
Nov 03 06:03:41.747982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:03:41.747948    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" path="/var/lib/kubelet/pods/48cf3c21-d4ce-469e-ab77-25cc5b979dd1/volumes"
Nov 03 06:03:43.692846 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:03:43.692776    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:04:07.967760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.967717    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:04:07.968410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.968389    4528 cpu_manager.go:414] "containerMap: removing container" podUID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" containerName="manager"
Nov 03 06:04:07.968410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.968411    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" containerName="manager"
Nov 03 06:04:07.968534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.968487    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="48cf3c21-d4ce-469e-ab77-25cc5b979dd1" containerName="manager"
Nov 03 06:04:07.972020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.971957    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:07.986467 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:07.986439    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-5c3542-openshift-default-dockercfg-mhnlv\""
Nov 03 06:04:08.002483 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.002450    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:04:08.068257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068222    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068282    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knlh4\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068379    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068417    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068439    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068498    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068623    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068736 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068701    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.068814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.068774    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.113969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.113930    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:04:08.122056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.122029    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.137285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.137236    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-df7236-openshift-default-dockercfg-qtf5v\""
Nov 03 06:04:08.141124 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.141088    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:04:08.170110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170065    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170137    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170300    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97tvq\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170333    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knlh4\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170364    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170397    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170428    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170461    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170485    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.170856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170519    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170546    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170592    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.170856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170598    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.171099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.170891    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.171474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.171453    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.171850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.171785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.172101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172080    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.172176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172122    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.172176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172148    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.172292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.172292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172227    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.172445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.172402    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.173884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.173861    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.175055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.175031    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.196103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.195646    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.196476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.196453    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knlh4\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.273905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.273809    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.273905    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.273932    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97tvq\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.273963    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.273995    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274207    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274670    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274692    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.274975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.274956    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.275628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.275358    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.275628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.275585    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.277473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.277406    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.281390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.281356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.287691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.287445    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:08.302543 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.302514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97tvq\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.303652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.303273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.438242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.438011    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:08.580665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.580621    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:04:08.585695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.585675    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.606389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.606357    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:04:08.611743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.611714    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-86728b-openshift-default-dockercfg-kswxd\""
Nov 03 06:04:08.618494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.618460    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:04:08.619711 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:08.619675    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d05fa7_9d57_46f6_a0ef_afea586291bd.slice/crio-d21ec1a8e9e43553517cdd4355807c2873d8c5f467a8baa2a54a56b5918f10c5 WatchSource:0}: Error finding container d21ec1a8e9e43553517cdd4355807c2873d8c5f467a8baa2a54a56b5918f10c5: Status 404 returned error can't find the container with id d21ec1a8e9e43553517cdd4355807c2873d8c5f467a8baa2a54a56b5918f10c5
Nov 03 06:04:08.622060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.622028    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.622160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.622089    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.622160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.622115    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.679051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679026    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679083    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679188 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679203    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679318    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679357    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679522    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.679576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.679552    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jkx\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781114    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781149    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781176    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781204    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jkx\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781323    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781372    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781419    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781453    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781602    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.781952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781906    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.782006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.781986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.782237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.782204    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.782361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.782269    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.784451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.784392    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.784451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.784422    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.817073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.817027    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:04:08.825923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.821775    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.826331 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.826293    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.826400 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.826382    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.826455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.826425    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:08.837876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.831954    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jkx\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.902596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.902569    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:08.942337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.936319    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:04:08.944535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.944507    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.960668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.960607    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-b7e447-openshift-default-dockercfg-slpt8\""
Nov 03 06:04:08.975468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.975435    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984549    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984588    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984624    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984652    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrt7\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984747    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984783    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:08.984966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:08.984851    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.085946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.085913    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.085957    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.085991    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086049    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrt7\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086112    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086176    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086232    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.086533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.086380    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.087604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.087395    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.087604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.087479    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.087604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.087549    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.087866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.087727    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.088095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.088049    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.089252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.089229    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.089592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.089571    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.110258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.110227    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrt7\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.110463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.110337    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.209599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.209550    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerStarted","Data":"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"}
Nov 03 06:04:09.209803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.209614    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerStarted","Data":"d21ec1a8e9e43553517cdd4355807c2873d8c5f467a8baa2a54a56b5918f10c5"}
Nov 03 06:04:09.211208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.211181    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerStarted","Data":"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"}
Nov 03 06:04:09.211311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.211226    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerStarted","Data":"e788322c3a90a2f64895d40e2642f6ce4e6d773ad07d5df279315dd02c401718"}
Nov 03 06:04:09.245962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.244202    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:04:09.247406 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:09.247363    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3144501e_3a21_4303_9bbc_afad4fc6dafa.slice/crio-19216113ab35b167fb3ff4840763c3830c7e0e320500a27d2c4e82d56936d248 WatchSource:0}: Error finding container 19216113ab35b167fb3ff4840763c3830c7e0e320500a27d2c4e82d56936d248: Status 404 returned error can't find the container with id 19216113ab35b167fb3ff4840763c3830c7e0e320500a27d2c4e82d56936d248
Nov 03 06:04:09.249941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.249903    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:09.250093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.250071    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:09.250188 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.250124    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:09.262525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.262467    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podStartSLOduration=2.262450996 podStartE2EDuration="2.262450996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:09.258902293 +0000 UTC m=+537.670299374" watchObservedRunningTime="2026-11-03 06:04:09.262450996 +0000 UTC m=+537.673848073"
Nov 03 06:04:09.288589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.288561    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:09.292694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.292674    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:09.293635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.293609    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:09.304569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.304516    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podStartSLOduration=1.304498398 podStartE2EDuration="1.304498398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:09.302262517 +0000 UTC m=+537.713659596" watchObservedRunningTime="2026-11-03 06:04:09.304498398 +0000 UTC m=+537.715895478"
Nov 03 06:04:09.443683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.443636    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:09.448432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.448401    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:09.604116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.604091    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:04:09.606621 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:09.606592    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e73abb8_51c6_406a_87d6_f090c3e5ac26.slice/crio-75eafb22fad5c6e8aa1fc46a0bb2953f506ecde76be399893dfc05a8845f35ea WatchSource:0}: Error finding container 75eafb22fad5c6e8aa1fc46a0bb2953f506ecde76be399893dfc05a8845f35ea: Status 404 returned error can't find the container with id 75eafb22fad5c6e8aa1fc46a0bb2953f506ecde76be399893dfc05a8845f35ea
Nov 03 06:04:09.609465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.609435    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:09.609552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.609518    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:09.609594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:09.609561    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:10.219370 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.219337    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerStarted","Data":"268034fc00ad8510d12d206023f2c4416dcb4ebd8b180a647f30de82e331b1a9"}
Nov 03 06:04:10.219871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.219395    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerStarted","Data":"19216113ab35b167fb3ff4840763c3830c7e0e320500a27d2c4e82d56936d248"}
Nov 03 06:04:10.221361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.221325    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerStarted","Data":"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"}
Nov 03 06:04:10.221496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.221381    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerStarted","Data":"75eafb22fad5c6e8aa1fc46a0bb2953f506ecde76be399893dfc05a8845f35ea"}
Nov 03 06:04:10.221876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.221844    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:10.222122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.222104    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:10.222654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.222632    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:10.222871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.222843    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:10.265435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.265383    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podStartSLOduration=2.265368422 podStartE2EDuration="2.265368422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:10.262680681 +0000 UTC m=+538.674077759" watchObservedRunningTime="2026-11-03 06:04:10.265368422 +0000 UTC m=+538.676765500"
Nov 03 06:04:10.293983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.293944    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:10.300457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.300426    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:10.351529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.351475    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podStartSLOduration=2.3514584640000002 podStartE2EDuration="2.351458464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:10.349270308 +0000 UTC m=+538.760667415" watchObservedRunningTime="2026-11-03 06:04:10.351458464 +0000 UTC m=+538.762855541"
Nov 03 06:04:10.902893 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.902849    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:10.908626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:10.908599    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:11.228495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:11.228394    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:11.228495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:11.228454    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:11.229381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:11.229358    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:11.229500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:11.229412    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:13.673771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.673734    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:04:13.679745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.679714    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.696091 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:04:13.696056    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:04:13.712304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.708283    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-df7236\""
Nov 03 06:04:13.712304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.708379    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-df7236-dockercfg-5zqgj\""
Nov 03 06:04:13.713010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.712980    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:04:13.741423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.741372    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdcr\" (UniqueName: \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.741677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.741446    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.774107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.774027    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:04:13.779709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.778725    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.810327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.810294    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:04:13.842235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.842202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cmfm\" (UniqueName: \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.842438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.842336    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.842506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.842465    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdcr\" (UniqueName: \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.842506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.842490    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.843184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.843160    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.869616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.869578    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdcr\" (UniqueName: \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\") pod \"payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:13.944161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.944070    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cmfm\" (UniqueName: \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.944319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.944175    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.944890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.944869    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.966683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.966648    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cmfm\" (UniqueName: \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\") pod \"payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:13.994616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:13.994577    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:14.111169 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:14.111135    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:14.293743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:14.293712    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:04:14.294584 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:14.294560    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf8416e0_a375_4239_a542_f8c84c98e548.slice/crio-0fa696c0e2eb3468b5ccce4ace376493b659c6f51a1b4444173e8fdc9361ae36 WatchSource:0}: Error finding container 0fa696c0e2eb3468b5ccce4ace376493b659c6f51a1b4444173e8fdc9361ae36: Status 404 returned error can't find the container with id 0fa696c0e2eb3468b5ccce4ace376493b659c6f51a1b4444173e8fdc9361ae36
Nov 03 06:04:14.410963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:14.410935    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:04:14.411821 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:14.411776    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26702801_3755_47d7_95ca_37207e03af26.slice/crio-c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37 WatchSource:0}: Error finding container c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37: Status 404 returned error can't find the container with id c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37
Nov 03 06:04:15.254628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.254589    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" event={"ID":"26702801-3755-47d7-95ca-37207e03af26","Type":"ContainerStarted","Data":"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"}
Nov 03 06:04:15.255210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.254654    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:15.255210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.254682    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" event={"ID":"26702801-3755-47d7-95ca-37207e03af26","Type":"ContainerStarted","Data":"c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37"}
Nov 03 06:04:15.256525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.256474    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" event={"ID":"df8416e0-a375-4239-a542-f8c84c98e548","Type":"ContainerStarted","Data":"91125266ea9ffc17ff9ecee8b2ea9cba70b121644bbf1eaba7932fe58b4c0f40"}
Nov 03 06:04:15.256525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.256515    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" event={"ID":"df8416e0-a375-4239-a542-f8c84c98e548","Type":"ContainerStarted","Data":"0fa696c0e2eb3468b5ccce4ace376493b659c6f51a1b4444173e8fdc9361ae36"}
Nov 03 06:04:15.256722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.256686    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:15.259603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.259582    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:04:15.263418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.263401    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.282040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.282005    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-5c3542\""
Nov 03 06:04:15.282170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.282144    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-5c3542-dockercfg-ctfkb\""
Nov 03 06:04:15.286080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.286054    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:04:15.335258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.335205    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" podStartSLOduration=2.3351917 podStartE2EDuration="2.3351917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:15.296000788 +0000 UTC m=+543.707397868" watchObservedRunningTime="2026-11-03 06:04:15.3351917 +0000 UTC m=+543.746588778"
Nov 03 06:04:15.337402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.337365    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" podStartSLOduration=2.337354358 podStartE2EDuration="2.337354358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:15.3347858 +0000 UTC m=+543.746182879" watchObservedRunningTime="2026-11-03 06:04:15.337354358 +0000 UTC m=+543.748751436"
Nov 03 06:04:15.360894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.360833    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.361140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.361021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfvz9\" (UniqueName: \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.361772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.361751    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:04:15.365625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.365610    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.383390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.383362    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:04:15.464018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.462274    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-862qj\" (UniqueName: \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.464018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.462362    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfvz9\" (UniqueName: \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.464018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.462528    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.464018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.462578    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.464018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.463962    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.503025 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.502982    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfvz9\" (UniqueName: \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\") pod \"payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.563756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.563722    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.564101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.563820    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-862qj\" (UniqueName: \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.564942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.564916    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.577076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.577042    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:15.610571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.610530    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-862qj\" (UniqueName: \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\") pod \"payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.679526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.679496    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:15.903768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:15.903730    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:04:15.905196 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:15.905164    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70b3b9d4_9ce2_4f40_9ea0_a56f7def3b5f.slice/crio-a6d434f6643b2ce9bf30ffab207a0c17d36efd7c63057fa9918b86ce31c47343 WatchSource:0}: Error finding container a6d434f6643b2ce9bf30ffab207a0c17d36efd7c63057fa9918b86ce31c47343: Status 404 returned error can't find the container with id a6d434f6643b2ce9bf30ffab207a0c17d36efd7c63057fa9918b86ce31c47343
Nov 03 06:04:16.037083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.037058    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:04:16.040029 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:16.040004    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831d6c36_a2ab_4013_b701_2206555582f5.slice/crio-5b9e5f56eeb223272d4f7a9a75b3b0a0ba8d3de394d6f460575c35e0a1208c06 WatchSource:0}: Error finding container 5b9e5f56eeb223272d4f7a9a75b3b0a0ba8d3de394d6f460575c35e0a1208c06: Status 404 returned error can't find the container with id 5b9e5f56eeb223272d4f7a9a75b3b0a0ba8d3de394d6f460575c35e0a1208c06
Nov 03 06:04:16.288992 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.288893    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" event={"ID":"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f","Type":"ContainerStarted","Data":"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"}
Nov 03 06:04:16.288992 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.288978    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" event={"ID":"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f","Type":"ContainerStarted","Data":"a6d434f6643b2ce9bf30ffab207a0c17d36efd7c63057fa9918b86ce31c47343"}
Nov 03 06:04:16.289446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.289126    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:16.290883 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.290854    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" event={"ID":"831d6c36-a2ab-4013-b701-2206555582f5","Type":"ContainerStarted","Data":"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"}
Nov 03 06:04:16.291012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.290900    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" event={"ID":"831d6c36-a2ab-4013-b701-2206555582f5","Type":"ContainerStarted","Data":"5b9e5f56eeb223272d4f7a9a75b3b0a0ba8d3de394d6f460575c35e0a1208c06"}
Nov 03 06:04:16.291084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.291029    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:16.329912 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.329859    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" podStartSLOduration=1.329844506 podStartE2EDuration="1.329844506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:16.32719172 +0000 UTC m=+544.738588796" watchObservedRunningTime="2026-11-03 06:04:16.329844506 +0000 UTC m=+544.741241584"
Nov 03 06:04:16.367615 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:16.367536    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" podStartSLOduration=1.367520834 podStartE2EDuration="1.367520834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:16.36649032 +0000 UTC m=+544.777887398" watchObservedRunningTime="2026-11-03 06:04:16.367520834 +0000 UTC m=+544.778917914"
Nov 03 06:04:17.566641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.566608    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:04:17.570524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.570507    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.591724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.591694    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:04:17.598343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.598316    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-9688d8\""
Nov 03 06:04:17.598521 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.598411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-9688d8-dockercfg-sqzwn\""
Nov 03 06:04:17.695828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.695777    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5jdx\" (UniqueName: \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.696027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.695913    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.796854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.796746    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5jdx\" (UniqueName: \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.797037 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.796906    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.797691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.797666    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.825859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.825773    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5jdx\" (UniqueName: \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\") pod \"payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:17.903819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:17.898570    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:18.218072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.218043    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:04:18.219276 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:18.219241    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789d295c_7bd4_4bb9_9f9d_bc56d6102849.slice/crio-0dce9aafd7c59dd537314042a57fa0756497d5de9458b6b63333f056c121e096 WatchSource:0}: Error finding container 0dce9aafd7c59dd537314042a57fa0756497d5de9458b6b63333f056c121e096: Status 404 returned error can't find the container with id 0dce9aafd7c59dd537314042a57fa0756497d5de9458b6b63333f056c121e096
Nov 03 06:04:18.305373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.305345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" event={"ID":"789d295c-7bd4-4bb9-9f9d-bc56d6102849","Type":"ContainerStarted","Data":"0dce9aafd7c59dd537314042a57fa0756497d5de9458b6b63333f056c121e096"}
Nov 03 06:04:18.486530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.486449    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:04:18.491307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.491283    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.509109 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.509073    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:04:18.518774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.515301    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-ba6892\""
Nov 03 06:04:18.518774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.515313    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-ba6892-dockercfg-d24pv\""
Nov 03 06:04:18.606708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.606670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.607291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.607264    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g878\" (UniqueName: \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.611594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.611567    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:04:18.618851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.618171    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.619009 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.618072    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:04:18.709240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.709202    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlj6\" (UniqueName: \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.709459 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.709296    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.709459 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.709423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.709589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.709466    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g878\" (UniqueName: \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.710150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.710127    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.732091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.732061    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g878\" (UniqueName: \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\") pod \"payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.810245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.810195    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:18.810596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.810571    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.810746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.810719    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlj6\" (UniqueName: \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.811640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.811615    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.835577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.835547    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlj6\" (UniqueName: \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\") pod \"payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:18.934349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:18.934313    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:19.162809 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.162323    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:04:19.164599 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:19.164571    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79d5a77_7809_456f_bf14_e8cf9fde05d7.slice/crio-0df0c6b5261c0a02b907ca610381019827fd0344ef32ad1c8d63a2d80af58439 WatchSource:0}: Error finding container 0df0c6b5261c0a02b907ca610381019827fd0344ef32ad1c8d63a2d80af58439: Status 404 returned error can't find the container with id 0df0c6b5261c0a02b907ca610381019827fd0344ef32ad1c8d63a2d80af58439
Nov 03 06:04:19.292697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.292608    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:04:19.352478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.352435    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" event={"ID":"b6ed1cc4-f120-43d1-8774-65bf446c151e","Type":"ContainerStarted","Data":"65a73fbb92731d225ce52f3578d9243feac270325fd47dca3de65f097eab6493"}
Nov 03 06:04:19.354712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.354634    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" event={"ID":"789d295c-7bd4-4bb9-9f9d-bc56d6102849","Type":"ContainerStarted","Data":"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"}
Nov 03 06:04:19.354939 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.354894    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:19.357560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.356849    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" event={"ID":"b79d5a77-7809-456f-bf14-e8cf9fde05d7","Type":"ContainerStarted","Data":"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"}
Nov 03 06:04:19.357560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.356893    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" event={"ID":"b79d5a77-7809-456f-bf14-e8cf9fde05d7","Type":"ContainerStarted","Data":"0df0c6b5261c0a02b907ca610381019827fd0344ef32ad1c8d63a2d80af58439"}
Nov 03 06:04:19.357560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.357469    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:19.447073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.447020    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" podStartSLOduration=1.447004449 podStartE2EDuration="1.447004449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:19.44552086 +0000 UTC m=+547.856917939" watchObservedRunningTime="2026-11-03 06:04:19.447004449 +0000 UTC m=+547.858401563"
Nov 03 06:04:19.449070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:19.449030    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" podStartSLOduration=2.449018106 podStartE2EDuration="2.449018106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:19.404532609 +0000 UTC m=+547.815929688" watchObservedRunningTime="2026-11-03 06:04:19.449018106 +0000 UTC m=+547.860415183"
Nov 03 06:04:20.249433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.249393    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:04:20.253486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.253465    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.304816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.301062    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-b7e447-dockercfg-2zpdx\""
Nov 03 06:04:20.307994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.307939    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:04:20.321493 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.321464    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-b7e447\""
Nov 03 06:04:20.338972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.338929    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8nz\" (UniqueName: \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.339177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.339052    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.389502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.389456    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" event={"ID":"b6ed1cc4-f120-43d1-8774-65bf446c151e","Type":"ContainerStarted","Data":"35efed43dbc1686a1a210db591b7f4639ef69447d23c7d67b58aba3cb8a9220e"}
Nov 03 06:04:20.406067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.405934    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:20.439733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.439697    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.439943 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.439847    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8nz\" (UniqueName: \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.440475 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.440455    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.448882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.448820    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" podStartSLOduration=2.448802066 podStartE2EDuration="2.448802066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:20.444684928 +0000 UTC m=+548.856082007" watchObservedRunningTime="2026-11-03 06:04:20.448802066 +0000 UTC m=+548.860199137"
Nov 03 06:04:20.465260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.465223    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8nz\" (UniqueName: \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\") pod \"payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.568399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.568361    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:20.907463 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:20.907427    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd85b3cc0_9251_4c7c_b47a_55338091aba3.slice/crio-6dd6dbd2c4599261f4b998f064726906e868bfff3a8eb6909b2f73b70a153856 WatchSource:0}: Error finding container 6dd6dbd2c4599261f4b998f064726906e868bfff3a8eb6909b2f73b70a153856: Status 404 returned error can't find the container with id 6dd6dbd2c4599261f4b998f064726906e868bfff3a8eb6909b2f73b70a153856
Nov 03 06:04:20.908934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:20.908906    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:04:21.407265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:21.407228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" event={"ID":"d85b3cc0-9251-4c7c-b47a-55338091aba3","Type":"ContainerStarted","Data":"f8f4ce8fee0507364f03470ac7d89c5479aec39c024fc847d1d4827d9bfdcea8"}
Nov 03 06:04:21.407757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:21.407288    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" event={"ID":"d85b3cc0-9251-4c7c-b47a-55338091aba3","Type":"ContainerStarted","Data":"6dd6dbd2c4599261f4b998f064726906e868bfff3a8eb6909b2f73b70a153856"}
Nov 03 06:04:21.407757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:21.407637    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:21.448180 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:21.448125    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" podStartSLOduration=1.448110363 podStartE2EDuration="1.448110363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:21.446690358 +0000 UTC m=+549.858087429" watchObservedRunningTime="2026-11-03 06:04:21.448110363 +0000 UTC m=+549.859507440"
Nov 03 06:04:22.863863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.863272    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:04:22.872910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.871468    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:04:22.872910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.871588    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:22.894819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.894136    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-86728b\""
Nov 03 06:04:22.915859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.898303    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-86728b-dockercfg-f2cqt\""
Nov 03 06:04:22.962094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.962045    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:22.962283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:22.962177    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svfpc\" (UniqueName: \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.063133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.063093    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svfpc\" (UniqueName: \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.063321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.063241    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.064215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.064180    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.112776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.112617    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svfpc\" (UniqueName: \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\") pod \"payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.230253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.230158    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:23.562813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:23.562769    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:04:23.563412 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:23.563385    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod498de3a8_6ecf_40fa_af5a_31845d02d5a8.slice/crio-58fe25715ab70d2bfed1afed81df967c9566afa14f19695b6127e27d2d303544 WatchSource:0}: Error finding container 58fe25715ab70d2bfed1afed81df967c9566afa14f19695b6127e27d2d303544: Status 404 returned error can't find the container with id 58fe25715ab70d2bfed1afed81df967c9566afa14f19695b6127e27d2d303544
Nov 03 06:04:24.432838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:24.432779    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" event={"ID":"498de3a8-6ecf-40fa-af5a-31845d02d5a8","Type":"ContainerStarted","Data":"63d47b1da8aa96de9ccd0911e6713aaa6949d1535492714d6293a83bb823519c"}
Nov 03 06:04:24.433210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:24.432864    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:24.433210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:24.432885    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" event={"ID":"498de3a8-6ecf-40fa-af5a-31845d02d5a8","Type":"ContainerStarted","Data":"58fe25715ab70d2bfed1afed81df967c9566afa14f19695b6127e27d2d303544"}
Nov 03 06:04:24.484889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:24.484822    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" podStartSLOduration=2.484802123 podStartE2EDuration="2.484802123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:24.479723291 +0000 UTC m=+552.891120369" watchObservedRunningTime="2026-11-03 06:04:24.484802123 +0000 UTC m=+552.896199194"
Nov 03 06:04:26.263989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:26.263938    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:04:26.264464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:26.264015    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:04:26.304818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:26.303970    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:04:26.305635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:26.305116    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:04:27.306124 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:27.306088    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:04:27.306641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:27.306178    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:04:29.473257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.473225    4528 generic.go:356] "Generic (PLEG): container finished" podID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerID="770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312" exitCode=0
Nov 03 06:04:29.473696 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.473302    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerDied","Data":"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"}
Nov 03 06:04:29.473751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.473736    4528 scope.go:122] "RemoveContainer" containerID="770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"
Nov 03 06:04:29.475168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.475134    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:29.475269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.475201    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:29.475269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:29.475235    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:30.425036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:30.424994    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:04:30.498252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:30.498208    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerStarted","Data":"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"}
Nov 03 06:04:31.411280 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:31.411244    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:04:31.438486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:31.438440    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:31.438705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:31.438684    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 06:04:31.438773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:31.438733    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 06:04:32.452493 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:32.452452    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 06:04:32.453123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:32.453092    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 06:04:32.454309 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:32.454156    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:04:32.454529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:32.454512    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:04:33.439386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:33.439343    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 06:04:33.439572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:33.439397    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 06:04:34.439303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:34.439262    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 06:04:34.440346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:34.439907    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 06:04:35.439403 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:35.439362    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body=
Nov 03 06:04:35.439831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:35.439422    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused"
Nov 03 06:04:35.442969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:35.442944    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:04:36.127504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.127468    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:04:36.137122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.135292    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.149549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.149507    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:04:36.151440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.150689    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"kube-root-ca.crt\""
Nov 03 06:04:36.151440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.151399    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"openshift-service-ca.crt\""
Nov 03 06:04:36.198282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.198218    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"simulator-w0-df7236-kserve-self-signed-certs\""
Nov 03 06:04:36.198730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.198688    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"default-dockercfg-9pvnm\""
Nov 03 06:04:36.231465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231419    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.231465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231464    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.231681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231627    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.231725 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231692    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82lsz\" (UniqueName: \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.231777 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231761    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.231863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.231805    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337373    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82lsz\" (UniqueName: \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337450    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337493    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337623    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.337754 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.337709    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.338429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.338394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.338618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.338601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.338821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.338768    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.340711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.340688    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.340976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.340955    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.363311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.363271    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82lsz\" (UniqueName: \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\") pod \"simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") " pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.442590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.442487    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:36.443034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.442814    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:36.443735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.443715    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:04:36.463480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.463442    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:36.817282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:36.817248    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:04:36.835929 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:36.835889    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podac8f80a9_d28a_4f4c_a0d8_13555623a606.slice/crio-1c15eb7b484946dc1917fb9ef2b6e14071aa6a969b36b244dd7ecf4dd68424e5 WatchSource:0}: Error finding container 1c15eb7b484946dc1917fb9ef2b6e14071aa6a969b36b244dd7ecf4dd68424e5: Status 404 returned error can't find the container with id 1c15eb7b484946dc1917fb9ef2b6e14071aa6a969b36b244dd7ecf4dd68424e5
Nov 03 06:04:37.652141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.652060    4528 generic.go:356] "Generic (PLEG): container finished" podID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerID="72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143" exitCode=0
Nov 03 06:04:37.652141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.652104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerDied","Data":"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"}
Nov 03 06:04:37.652654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.652535    4528 scope.go:122] "RemoveContainer" containerID="72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"
Nov 03 06:04:37.653979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.653946    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:37.654099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.654028    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:37.654099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.654061    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:37.654223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.654103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" event={"ID":"ac8f80a9-d28a-4f4c-a0d8-13555623a606","Type":"ContainerStarted","Data":"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"}
Nov 03 06:04:37.654223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.654135    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" event={"ID":"ac8f80a9-d28a-4f4c-a0d8-13555623a606","Type":"ContainerStarted","Data":"1c15eb7b484946dc1917fb9ef2b6e14071aa6a969b36b244dd7ecf4dd68424e5"}
Nov 03 06:04:37.758092 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:37.757650    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podStartSLOduration=1.757623847 podStartE2EDuration="1.757623847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:37.757329172 +0000 UTC m=+566.168726260" watchObservedRunningTime="2026-11-03 06:04:37.757623847 +0000 UTC m=+566.169020928"
Nov 03 06:04:38.463750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:38.463709    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:38.662334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:38.662290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerStarted","Data":"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"}
Nov 03 06:04:39.291930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:39.291229    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:39.291930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:39.291394    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 06:04:39.291930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:39.291439    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 06:04:40.227815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:40.223918    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:40.299816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:40.299220    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 06:04:40.299816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:40.299286    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 06:04:41.293819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:41.292153    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 06:04:41.293819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:41.292227    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 06:04:42.272982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.272941    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:04:42.273198 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.273000    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:04:42.298819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.297774    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 06:04:42.298819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.297861    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 06:04:42.740333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.740243    4528 generic.go:356] "Generic (PLEG): container finished" podID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerID="f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352" exitCode=0
Nov 03 06:04:42.740558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.740535    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerDied","Data":"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"}
Nov 03 06:04:42.741166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.741141    4528 scope.go:122] "RemoveContainer" containerID="f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"
Nov 03 06:04:42.742952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.742919    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:42.743046 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.743003    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:42.743088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:42.743049    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:43.299913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:43.293224    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Nov 03 06:04:43.299913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:43.293286    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Nov 03 06:04:43.942903 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:04:43.942861    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:04:43.945224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:43.945193    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerStarted","Data":"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"}
Nov 03 06:04:44.301295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:44.301202    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:44.302018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:44.301911    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:44.302197 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:44.302172    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:44.342502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:44.342199    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:44.344317 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:44.344291    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:04:45.293582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:45.293546    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:45.293907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:45.293880    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:46.308065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:46.308031    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:46.308622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:46.308414    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:46.464400 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:46.464359    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:46.507680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:46.507519    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:46.986576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:46.986541    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:04:47.303756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:47.303722    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:47.304079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:47.304050    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:48.295818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:48.295253    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:48.295818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:48.295337    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:49.294076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:49.294032    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body=
Nov 03 06:04:49.294262 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:49.294120    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused"
Nov 03 06:04:50.297258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:50.297221    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:50.298694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:50.297643    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:50.298694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:50.298247    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:04:51.023134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.023095    4528 generic.go:356] "Generic (PLEG): container finished" podID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerID="268034fc00ad8510d12d206023f2c4416dcb4ebd8b180a647f30de82e331b1a9" exitCode=0
Nov 03 06:04:51.023332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.023174    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerDied","Data":"268034fc00ad8510d12d206023f2c4416dcb4ebd8b180a647f30de82e331b1a9"}
Nov 03 06:04:51.023713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.023693    4528 scope.go:122] "RemoveContainer" containerID="268034fc00ad8510d12d206023f2c4416dcb4ebd8b180a647f30de82e331b1a9"
Nov 03 06:04:51.025133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.025101    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:51.025218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.025168    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:51.025218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:51.025204    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:04:52.032233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:52.032168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerStarted","Data":"1e2721d0ba3e55a5b29a206b883a1bbbfe09bd2d5ad63e74af14ae8797a1f0fe"}
Nov 03 06:04:52.903255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:52.903162    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:52.905251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:52.904891    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 06:04:52.905251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:52.904967    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 06:04:53.903878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:53.903245    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 06:04:53.903878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:53.903314    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 06:04:54.903828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:54.903747    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 06:04:54.904011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:54.903883    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 06:04:55.903464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:55.903412    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 06:04:55.903662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:55.903512    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 06:04:56.228832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:56.228704    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:56.903845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:56.903772    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Nov 03 06:04:56.904061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:56.903870    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Nov 03 06:04:57.907901 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:57.907867    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:57.909260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:57.909235    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:04:58.019193 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.019152    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:04:58.023097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.023066    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.041542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.041504    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:04:58.043582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.043548    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"kube-root-ca.crt\""
Nov 03 06:04:58.044721 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.044697    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"default-dockercfg-v45nf\""
Nov 03 06:04:58.044960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.044833    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"simulator-w5-ba6892-kserve-self-signed-certs\""
Nov 03 06:04:58.044960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.044954    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"openshift-service-ca.crt\""
Nov 03 06:04:58.060950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.060905    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.061126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.060975    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.061169 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.061122    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.061213 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.061175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.061324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.061300    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrrw6\" (UniqueName: \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.061402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.061339    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162247    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162299    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162550    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162649    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162706    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162732    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162735    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrrw6\" (UniqueName: \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.162962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.162826    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.163053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.163027    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.164813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.164777    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.165295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.165273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.186611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.186577    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrrw6\" (UniqueName: \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\") pod \"simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") " pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.338752 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.338703    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:04:58.644038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:58.643999    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:04:58.645303 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:04:58.645276    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7089559_f797_4ce8_bab2_828a446aee78.slice/crio-31580a815f714aa12c2e327750aa997936375738bc11e14690de66e597be762b WatchSource:0}: Error finding container 31580a815f714aa12c2e327750aa997936375738bc11e14690de66e597be762b: Status 404 returned error can't find the container with id 31580a815f714aa12c2e327750aa997936375738bc11e14690de66e597be762b
Nov 03 06:04:59.091463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:59.091418    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" event={"ID":"b7089559-f797-4ce8-bab2-828a446aee78","Type":"ContainerStarted","Data":"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"}
Nov 03 06:04:59.091885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:59.091470    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" event={"ID":"b7089559-f797-4ce8-bab2-828a446aee78","Type":"ContainerStarted","Data":"31580a815f714aa12c2e327750aa997936375738bc11e14690de66e597be762b"}
Nov 03 06:04:59.132627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:59.132560    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podStartSLOduration=2.132542054 podStartE2EDuration="2.132542054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:04:59.131971359 +0000 UTC m=+587.543368437" watchObservedRunningTime="2026-11-03 06:04:59.132542054 +0000 UTC m=+587.543939133"
Nov 03 06:04:59.339159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:04:59.339119    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:05:04.964270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.964154    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:05:04.968403 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.968375    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:04.988120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.988075    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:05:04.994433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.994387    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"simulator-w6-86728b-kserve-self-signed-certs\""
Nov 03 06:05:04.994668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.994642    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"default-dockercfg-r27rx\""
Nov 03 06:05:04.994957 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.994936    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"openshift-service-ca.crt\""
Nov 03 06:05:04.995043 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:04.994995    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-86728b\"/\"kube-root-ca.crt\""
Nov 03 06:05:05.055936 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.055898    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.056147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.055969    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.056147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.056085    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8v5m5\" (UniqueName: \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.056271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.056150    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.056271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.056249    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.056368 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.056298    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157044    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157104    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157154    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8v5m5\" (UniqueName: \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157292    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157371    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157429    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157549    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157612    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.157735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.157713    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.159488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.159463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.159989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.159966    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.180370 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.180327    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8v5m5\" (UniqueName: \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\") pod \"simulator-w6-86728b-kserve-75d75d6bbb-w8687\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") " pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.283500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.283402    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:05.589730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:05.589694    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:05:05.590169 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:05:05.590142    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podccfca8f4_b49c_4c85_9e65_72548e8a5cbf.slice/crio-014628b4dd95e03b22378f278e95139e4ca7a6f50339b9a2a85dc0ed6a5f0a6c WatchSource:0}: Error finding container 014628b4dd95e03b22378f278e95139e4ca7a6f50339b9a2a85dc0ed6a5f0a6c: Status 404 returned error can't find the container with id 014628b4dd95e03b22378f278e95139e4ca7a6f50339b9a2a85dc0ed6a5f0a6c
Nov 03 06:05:06.144848 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:06.144812    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" event={"ID":"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf","Type":"ContainerStarted","Data":"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"}
Nov 03 06:05:06.145363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:06.144855    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" event={"ID":"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf","Type":"ContainerStarted","Data":"014628b4dd95e03b22378f278e95139e4ca7a6f50339b9a2a85dc0ed6a5f0a6c"}
Nov 03 06:05:06.185696 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:06.185640    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podStartSLOduration=2.185623916 podStartE2EDuration="2.185623916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:06.184850496 +0000 UTC m=+594.596247573" watchObservedRunningTime="2026-11-03 06:05:06.185623916 +0000 UTC m=+594.597020994"
Nov 03 06:05:06.284528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:06.284486    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:08.339450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:08.339401    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:05:08.352380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:08.352353    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:05:09.176906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:09.176875    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:05:13.694171 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:05:13.694130    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:05:15.284390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:15.284346    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:15.298327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:15.298289    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:16.229671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:16.229637    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:05:19.554060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.554011    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:05:19.558858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.558828    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:05:19.559008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.558935    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.573185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.573157    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-df7236\"/\"premium-w0-df7236-kserve-self-signed-certs\""
Nov 03 06:05:19.631127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.631127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631130    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.631465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.631465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rvbr\" (UniqueName: \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.631465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631391    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.631609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.631513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.733475 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.732728    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.733975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.733945    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.734900 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.734875    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.735252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.735233    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.735537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.735518    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rvbr\" (UniqueName: \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.736335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.736315    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.755921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.744007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.757294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.756900    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.757294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.757241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.768776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.764114    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.768776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.766133    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.768776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.767209    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rvbr\" (UniqueName: \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\") pod \"premium-w0-df7236-kserve-7599955768-5gkhp\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") " pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:19.871316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:19.871208    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:20.184323 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:20.184293    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:05:20.185035 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:05:20.185000    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5b698bb_3674_490d_bbd2_62fbd4b7497e.slice/crio-8203d26d0e27ff7c666df79978cc606395b0aa07cffc658857a72396b4b1be38 WatchSource:0}: Error finding container 8203d26d0e27ff7c666df79978cc606395b0aa07cffc658857a72396b4b1be38: Status 404 returned error can't find the container with id 8203d26d0e27ff7c666df79978cc606395b0aa07cffc658857a72396b4b1be38
Nov 03 06:05:20.253113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:20.253077    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" event={"ID":"a5b698bb-3674-490d-bbd2-62fbd4b7497e","Type":"ContainerStarted","Data":"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"}
Nov 03 06:05:20.253301 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:20.253127    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" event={"ID":"a5b698bb-3674-490d-bbd2-62fbd4b7497e","Type":"ContainerStarted","Data":"8203d26d0e27ff7c666df79978cc606395b0aa07cffc658857a72396b4b1be38"}
Nov 03 06:05:20.292988 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:20.292918    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podStartSLOduration=1.292903414 podStartE2EDuration="1.292903414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:20.292427942 +0000 UTC m=+608.703825032" watchObservedRunningTime="2026-11-03 06:05:20.292903414 +0000 UTC m=+608.704300492"
Nov 03 06:05:20.872372 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:20.872334    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:29.871777 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:29.871731    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:29.885531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:29.885499    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:30.368144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:30.368115    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:05:43.696374 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:05:43.696333    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:05:54.153225 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.153188    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:05:54.158233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.158210    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.175113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.175072    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"kube-root-ca.crt\""
Nov 03 06:05:54.186640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.186608    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"openshift-service-ca.crt\""
Nov 03 06:05:54.186824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.186677    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"premium-w3-b7e447-kserve-self-signed-certs\""
Nov 03 06:05:54.186824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.186769    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-b7e447\"/\"default-dockercfg-xxndc\""
Nov 03 06:05:54.191316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.191284    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:05:54.253611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.253807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253626    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.253807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253718    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.253807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.253807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253806    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.254017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.253918    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpsf9\" (UniqueName: \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.358568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358526    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.358840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358609    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.358840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358645    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.358840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358683    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpsf9\" (UniqueName: \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.359064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358838    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.359064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.358910    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.359401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.359376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.359669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.359644    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.364174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.361164    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.367551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.367522    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.370465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.370420    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.394530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.394491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpsf9\" (UniqueName: \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\") pod \"premium-w3-b7e447-kserve-869b76c5d-qs422\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") " pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.472571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.472489    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:05:54.775495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:54.775465    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:05:54.776405 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:05:54.776379    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c1d658e_c4c7_425e_9b83_b3431fbcff06.slice/crio-5217a017c725324eb7a4994e0a6afa20b0e0f101d725aad84dd3e908718c7850 WatchSource:0}: Error finding container 5217a017c725324eb7a4994e0a6afa20b0e0f101d725aad84dd3e908718c7850: Status 404 returned error can't find the container with id 5217a017c725324eb7a4994e0a6afa20b0e0f101d725aad84dd3e908718c7850
Nov 03 06:05:55.545035 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:55.544997    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" event={"ID":"1c1d658e-c4c7-425e-9b83-b3431fbcff06","Type":"ContainerStarted","Data":"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"}
Nov 03 06:05:55.545427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:55.545048    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" event={"ID":"1c1d658e-c4c7-425e-9b83-b3431fbcff06","Type":"ContainerStarted","Data":"5217a017c725324eb7a4994e0a6afa20b0e0f101d725aad84dd3e908718c7850"}
Nov 03 06:05:55.587726 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:55.587662    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podStartSLOduration=1.5876464160000001 podStartE2EDuration="1.587646416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:05:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:05:55.584636604 +0000 UTC m=+643.996033683" watchObservedRunningTime="2026-11-03 06:05:55.587646416 +0000 UTC m=+643.999043494"
Nov 03 06:05:56.473297 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:05:56.473261    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:06:04.473008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:04.472975    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:06:04.485758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:04.485732    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:06:04.628980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:04.628951    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:06:13.694687 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:06:13.694639    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:06:16.515705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.515666    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"]
Nov 03 06:06:16.519586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.519566    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.522412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.522388    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.522541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.522454    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.522595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.522574    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crkll\" (UniqueName: \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.538587 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.538558    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"]
Nov 03 06:06:16.544573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.544552    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:06:16.544731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.544707    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:06:16.544821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.544643    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:06:16.545296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.545273    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:06:16.623571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.623537    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crkll\" (UniqueName: \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.623740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.623616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.623740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.623672    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.624385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.624360    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.625989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.625964    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.646534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.646509    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crkll\" (UniqueName: \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\") pod \"maas-api-revoke-sub-7b35c084-xkvcd\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.837697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.837666    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:16.930975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.930936    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"]
Nov 03 06:06:16.935219 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.935199    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:16.957448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:16.957410    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"]
Nov 03 06:06:17.028870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.028836    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbcgm\" (UniqueName: \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.029036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.028902    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.029036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.028972    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.130470 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.130391    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbcgm\" (UniqueName: \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.130470 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.130456    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.130652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.130489    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.131163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.131141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.132880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.132850    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.142732 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:17.142710    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9a89d27_9c98_4e2f_8271_a5898257d53f.slice/crio-40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873 WatchSource:0}: Error finding container 40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873: Status 404 returned error can't find the container with id 40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873
Nov 03 06:06:17.144240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.144198    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"]
Nov 03 06:06:17.153280 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.153259    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbcgm\" (UniqueName: \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\") pod \"maas-api-revoke-sub-39cae823-j45zw\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.253743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.253709    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:17.556545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.556512    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"]
Nov 03 06:06:17.557236 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:17.557198    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5a3755a_8c97_4480_806b_703483a7b5c5.slice/crio-0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f WatchSource:0}: Error finding container 0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f: Status 404 returned error can't find the container with id 0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f
Nov 03 06:06:17.712137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.712085    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd" event={"ID":"c9a89d27-9c98-4e2f-8271-a5898257d53f","Type":"ContainerStarted","Data":"40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873"}
Nov 03 06:06:17.713668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:17.713637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw" event={"ID":"d5a3755a-8c97-4480-806b-703483a7b5c5","Type":"ContainerStarted","Data":"0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f"}
Nov 03 06:06:20.749813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:20.749762    4528 generic.go:356] "Generic (PLEG): container finished" podID="c9a89d27-9c98-4e2f-8271-a5898257d53f" containerID="52f85df3faa7a019439ac6d682a53d3d36c541eae95d01934f54be93bb0101d6" exitCode=0
Nov 03 06:06:20.750303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:20.749891    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd" event={"ID":"c9a89d27-9c98-4e2f-8271-a5898257d53f","Type":"ContainerDied","Data":"52f85df3faa7a019439ac6d682a53d3d36c541eae95d01934f54be93bb0101d6"}
Nov 03 06:06:20.752657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:20.752616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw" event={"ID":"d5a3755a-8c97-4480-806b-703483a7b5c5","Type":"ContainerStarted","Data":"d4511a7589c490247d38100fc4951cf372178c9c7e81041d3b798efdf44b1fd9"}
Nov 03 06:06:20.834234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:20.834165    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw" podStartSLOduration=2.45732938 podStartE2EDuration="4.834145248s" totalImagesPullingTime="2.376815868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:20.830125156 +0000 UTC m=+669.241522235" watchObservedRunningTime="2026-11-03 06:06:20.834145248 +0000 UTC m=+669.245542327"
Nov 03 06:06:21.763756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:21.763717    4528 generic.go:356] "Generic (PLEG): container finished" podID="d5a3755a-8c97-4480-806b-703483a7b5c5" containerID="d4511a7589c490247d38100fc4951cf372178c9c7e81041d3b798efdf44b1fd9" exitCode=0
Nov 03 06:06:21.764237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:21.763814    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw" event={"ID":"d5a3755a-8c97-4480-806b-703483a7b5c5","Type":"ContainerDied","Data":"d4511a7589c490247d38100fc4951cf372178c9c7e81041d3b798efdf44b1fd9"}
Nov 03 06:06:21.932458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:21.932432    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:22.013981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.013897    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\") pod \"c9a89d27-9c98-4e2f-8271-a5898257d53f\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") "
Nov 03 06:06:22.013981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.013975    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\") pod \"c9a89d27-9c98-4e2f-8271-a5898257d53f\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") "
Nov 03 06:06:22.014209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.014137    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\" (UniqueName: \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\") pod \"c9a89d27-9c98-4e2f-8271-a5898257d53f\" (UID: \"c9a89d27-9c98-4e2f-8271-a5898257d53f\") "
Nov 03 06:06:22.014318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.014293    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca" pod "c9a89d27-9c98-4e2f-8271-a5898257d53f" (UID: "c9a89d27-9c98-4e2f-8271-a5898257d53f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:22.014672 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.014648    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:22.016362 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.016332    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca" pod "c9a89d27-9c98-4e2f-8271-a5898257d53f" (UID: "c9a89d27-9c98-4e2f-8271-a5898257d53f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:22.016470 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.016401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll" pod "c9a89d27-9c98-4e2f-8271-a5898257d53f" (UID: "c9a89d27-9c98-4e2f-8271-a5898257d53f"). InnerVolumeSpecName "kube-api-access-crkll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:22.115690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.115656    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9a89d27-9c98-4e2f-8271-a5898257d53f-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:22.115690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.115686    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crkll\" (UniqueName: \"kubernetes.io/projected/c9a89d27-9c98-4e2f-8271-a5898257d53f-kube-api-access-crkll\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:22.775185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.775158    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"
Nov 03 06:06:22.775185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.775168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd" event={"ID":"c9a89d27-9c98-4e2f-8271-a5898257d53f","Type":"ContainerDied","Data":"40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873"}
Nov 03 06:06:22.775624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.775229    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40089a952419f497eeb724486761291e72b443baaea59ea8d20731fc6219b873"
Nov 03 06:06:22.932787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:22.932753    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:23.026040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.025942    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\") pod \"d5a3755a-8c97-4480-806b-703483a7b5c5\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") "
Nov 03 06:06:23.026222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.026082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\") pod \"d5a3755a-8c97-4480-806b-703483a7b5c5\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") "
Nov 03 06:06:23.026222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.026151    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\" (UniqueName: \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\") pod \"d5a3755a-8c97-4480-806b-703483a7b5c5\" (UID: \"d5a3755a-8c97-4480-806b-703483a7b5c5\") "
Nov 03 06:06:23.026481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.026453    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca" pod "d5a3755a-8c97-4480-806b-703483a7b5c5" (UID: "d5a3755a-8c97-4480-806b-703483a7b5c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:23.026704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.026687    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:23.028332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.028305    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca" pod "d5a3755a-8c97-4480-806b-703483a7b5c5" (UID: "d5a3755a-8c97-4480-806b-703483a7b5c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:23.028332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.028321    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm" pod "d5a3755a-8c97-4480-806b-703483a7b5c5" (UID: "d5a3755a-8c97-4480-806b-703483a7b5c5"). InnerVolumeSpecName "kube-api-access-kbcgm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:23.127406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.127362    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d5a3755a-8c97-4480-806b-703483a7b5c5-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:23.127406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.127395    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbcgm\" (UniqueName: \"kubernetes.io/projected/d5a3755a-8c97-4480-806b-703483a7b5c5-kube-api-access-kbcgm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:23.691038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.690956    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:06:23.691485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691472    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d5a3755a-8c97-4480-806b-703483a7b5c5" containerName="revoke-keys"
Nov 03 06:06:23.691542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691486    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5a3755a-8c97-4480-806b-703483a7b5c5" containerName="revoke-keys"
Nov 03 06:06:23.691542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691529    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c9a89d27-9c98-4e2f-8271-a5898257d53f" containerName="revoke-keys"
Nov 03 06:06:23.691542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691535    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9a89d27-9c98-4e2f-8271-a5898257d53f" containerName="revoke-keys"
Nov 03 06:06:23.691636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691613    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9a89d27-9c98-4e2f-8271-a5898257d53f" containerName="revoke-keys"
Nov 03 06:06:23.691636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.691622    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5a3755a-8c97-4480-806b-703483a7b5c5" containerName="revoke-keys"
Nov 03 06:06:23.695335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.695314    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.709451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.709428    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-9688d8-openshift-default-dockercfg-ldnhj\""
Nov 03 06:06:23.725595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.725559    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:06:23.782525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.782497    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"
Nov 03 06:06:23.783038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.782495    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw" event={"ID":"d5a3755a-8c97-4480-806b-703483a7b5c5","Type":"ContainerDied","Data":"0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f"}
Nov 03 06:06:23.783038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.782634    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0350d3f07df1ac2ca3c556e8bf970e097d2a356983eeee2abfe1a76dad85d52f"
Nov 03 06:06:23.838098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838064    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838273 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838119    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxfxj\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838273 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838157    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838296    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838325    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838367    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838450    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838484    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.838605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.838564    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939231    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939275    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939357    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939418    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxfxj\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939552    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939598    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.939886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.939787    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.940150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.940124    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.940306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.940126    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.940306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.940275    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.940832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.940780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.941877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.941825    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.942167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.942144    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.962675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.962640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:23.962879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:23.962806    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxfxj\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:24.010425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.010387    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:24.107607 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.107540    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:06:24.125920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.125888    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.140831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.140785    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:06:24.197773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.197688    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-ba6892-openshift-default-dockercfg-mtf55\""
Nov 03 06:06:24.259162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259126    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259339 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259170    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259339 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259252    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259339 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259283    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259423    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259473    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259577    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259606    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.259687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.259674    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgh2\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.355073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.355047    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:06:24.355726 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:24.355703    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cebc3ba_dad7_4210_bea9_0bd43834b6af.slice/crio-14ca1742475c164b1e8599f2fc6cdfd32ebaa3788de38a522fde77a6ffd17ab7 WatchSource:0}: Error finding container 14ca1742475c164b1e8599f2fc6cdfd32ebaa3788de38a522fde77a6ffd17ab7: Status 404 returned error can't find the container with id 14ca1742475c164b1e8599f2fc6cdfd32ebaa3788de38a522fde77a6ffd17ab7
Nov 03 06:06:24.358356 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.358320    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.358455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.358386    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.358455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.358413    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.360724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360699    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.360845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360740    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.360845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360786    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.360845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360838    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360885    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgh2\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360917    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360945    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.360970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.361024    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.361650    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.361918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.361901    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.362584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.362563    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.362820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.362773    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.363194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.363161    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.363588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.363567    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.365180 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.365162    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.384361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.384339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.384662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.384639    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgh2\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.448882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.448786    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:24.780027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.779950    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:06:24.793688 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:24.793652    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2e49f45_2645_437a_8b7e_516634115788.slice/crio-21a269a59a7867a997cdc812f641ffd3e06acd32e8fd5076776c1ab5145e9dc7 WatchSource:0}: Error finding container 21a269a59a7867a997cdc812f641ffd3e06acd32e8fd5076776c1ab5145e9dc7: Status 404 returned error can't find the container with id 21a269a59a7867a997cdc812f641ffd3e06acd32e8fd5076776c1ab5145e9dc7
Nov 03 06:06:24.795534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.795403    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" event={"ID":"6cebc3ba-dad7-4210-bea9-0bd43834b6af","Type":"ContainerStarted","Data":"81813a202d47385b67754eb4c58ec5ae6b469c6e861c79e2cafcc4c6c0045213"}
Nov 03 06:06:24.795534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.795464    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" event={"ID":"6cebc3ba-dad7-4210-bea9-0bd43834b6af","Type":"ContainerStarted","Data":"14ca1742475c164b1e8599f2fc6cdfd32ebaa3788de38a522fde77a6ffd17ab7"}
Nov 03 06:06:24.796491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.796447    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.796605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.796535    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.796605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.796576    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:06:24.839859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:24.839789    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podStartSLOduration=1.839772333 podStartE2EDuration="1.839772333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:24.838356908 +0000 UTC m=+673.249753987" watchObservedRunningTime="2026-11-03 06:06:24.839772333 +0000 UTC m=+673.251169410"
Nov 03 06:06:25.011467 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.011378    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:25.013357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.013331    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:25.013581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.013550    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:25.845540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.845439    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" event={"ID":"f2e49f45-2645-437a-8b7e-516634115788","Type":"ContainerStarted","Data":"8012a8a6c59d673a8d519f51d7b201bb47f19a7e98a1979c4b4c85ed105ec234"}
Nov 03 06:06:25.845540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.845489    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" event={"ID":"f2e49f45-2645-437a-8b7e-516634115788","Type":"ContainerStarted","Data":"21a269a59a7867a997cdc812f641ffd3e06acd32e8fd5076776c1ab5145e9dc7"}
Nov 03 06:06:25.933817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:25.928215    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podStartSLOduration=1.928191799 podStartE2EDuration="1.928191799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:25.905258369 +0000 UTC m=+674.316655460" watchObservedRunningTime="2026-11-03 06:06:25.928191799 +0000 UTC m=+674.339588873"
Nov 03 06:06:26.014297 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:26.014257    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:26.014580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:26.014554    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:26.449579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:26.449536    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:26.450910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:26.450888    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:26.450977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:26.450931    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:27.018819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:27.016749    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:27.018819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:27.016842    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:27.449309 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:27.449267    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:27.449501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:27.449324    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:28.021818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:28.017213    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:28.021818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:28.017279    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:28.449480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:28.449416    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:28.449653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:28.449564    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:29.014819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:29.011211    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:29.014819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:29.011296    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:29.449143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:29.449107    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:29.449549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:29.449168    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:30.018819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:30.018239    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:30.018819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:30.018321    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:30.449657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:30.449620    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:30.450084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:30.449701    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:31.010927 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.010890    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body=
Nov 03 06:06:31.011136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.010957    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused"
Nov 03 06:06:31.088048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.087994    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"]
Nov 03 06:06:31.092108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.092078    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.109127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.108627    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"]
Nov 03 06:06:31.114513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.114271    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:06:31.114513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.114411    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:06:31.114513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.114467    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:06:31.118189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.118151    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:06:31.200740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.200699    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfgd7\" (UniqueName: \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.200961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.200899    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.201027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.201002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.302153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.302121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfgd7\" (UniqueName: \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.302356 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.302234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.302356 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.302259    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.303058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.303034    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.304904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.304871    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.327336 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.327299    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfgd7\" (UniqueName: \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\") pod \"maas-api-revoke-sub-705e06bf-htm4b\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.408483 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.408438    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:31.449195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.449158    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Nov 03 06:06:31.449371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.449224    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Nov 03 06:06:31.737660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.737110    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"]
Nov 03 06:06:31.746837 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:31.742034    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a70bce6_ee8b_4fd5_9400_d8f8944926d8.slice/crio-e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836 WatchSource:0}: Error finding container e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836: Status 404 returned error can't find the container with id e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836
Nov 03 06:06:31.931329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:31.931290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b" event={"ID":"9a70bce6-ee8b-4fd5-9400-d8f8944926d8","Type":"ContainerStarted","Data":"e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836"}
Nov 03 06:06:32.015154 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.015125    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:32.015502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.015480    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:32.016174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.016154    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:06:32.126115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.126079    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"]
Nov 03 06:06:32.130549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.130522    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.148904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.148866    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"]
Nov 03 06:06:32.314729 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.314697    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmgc6\" (UniqueName: \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.314908 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.314750    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.314908 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.314867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.416068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.416030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.416253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.416179    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmgc6\" (UniqueName: \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.416253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.416228    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.416774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.416737    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.418862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.418841    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.439272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.439242    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmgc6\" (UniqueName: \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\") pod \"maas-api-revoke-sub-e5dffa33-bntvv\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.451335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.451307    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:32.454625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.454602    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:32.455184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.455092    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:32.456001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.455982    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:06:32.725617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.725510    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"]
Nov 03 06:06:32.730084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.730062    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.748575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.748541    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"]
Nov 03 06:06:32.756008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.755977    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"]
Nov 03 06:06:32.760003 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:32.759974    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1402809e_9c66_445b_8d96_addd2cfed64f.slice/crio-0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9 WatchSource:0}: Error finding container 0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9: Status 404 returned error can't find the container with id 0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9
Nov 03 06:06:32.823248 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.823216    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.823422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.823357    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.823460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.823420    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdr9x\" (UniqueName: \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.924492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.924441    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.924690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.924522    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdr9x\" (UniqueName: \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.924762 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.924726    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.925327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.925270    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.927407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.927385    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:32.939844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.939818    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv" event={"ID":"1402809e-9c66-445b-8d96-addd2cfed64f","Type":"ContainerStarted","Data":"0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9"}
Nov 03 06:06:32.941344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.941318    4528 generic.go:356] "Generic (PLEG): container finished" podID="9a70bce6-ee8b-4fd5-9400-d8f8944926d8" containerID="67946309c4d902bd1b814a555d2e32d48cd4163394e064d4bf10bf1410d15759" exitCode=0
Nov 03 06:06:32.941456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.941354    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b" event={"ID":"9a70bce6-ee8b-4fd5-9400-d8f8944926d8","Type":"ContainerDied","Data":"67946309c4d902bd1b814a555d2e32d48cd4163394e064d4bf10bf1410d15759"}
Nov 03 06:06:32.948610 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:32.948585    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdr9x\" (UniqueName: \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\") pod \"maas-api-revoke-sub-085a34ca-mz9fw\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:33.046473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.046440    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:33.428417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.428380    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"]
Nov 03 06:06:33.428838 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:33.428807    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2d3b31_7a0e_40ae_a1b8_33b588b593d2.slice/crio-b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08 WatchSource:0}: Error finding container b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08: Status 404 returned error can't find the container with id b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08
Nov 03 06:06:33.950010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.949923    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw" event={"ID":"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2","Type":"ContainerStarted","Data":"4fea7f61c8e07859e07ba966f6a9a0438ce3066f687a1ffc6e20f9e10e78627e"}
Nov 03 06:06:33.950010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.949983    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw" event={"ID":"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2","Type":"ContainerStarted","Data":"b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08"}
Nov 03 06:06:33.951540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.951514    4528 generic.go:356] "Generic (PLEG): container finished" podID="1402809e-9c66-445b-8d96-addd2cfed64f" containerID="39370318a6458dcac71623af6ad0f20a6600faf9844807568feaedd796ec1665" exitCode=0
Nov 03 06:06:33.951639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.951591    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv" event={"ID":"1402809e-9c66-445b-8d96-addd2cfed64f","Type":"ContainerDied","Data":"39370318a6458dcac71623af6ad0f20a6600faf9844807568feaedd796ec1665"}
Nov 03 06:06:33.989938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:33.989885    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw" podStartSLOduration=1.98987102 podStartE2EDuration="1.98987102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:33.989067783 +0000 UTC m=+682.400464862" watchObservedRunningTime="2026-11-03 06:06:33.98987102 +0000 UTC m=+682.401268133"
Nov 03 06:06:34.082945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.082919    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:34.140715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.140674    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\") pod \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") "
Nov 03 06:06:34.140911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.140757    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\" (UniqueName: \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\") pod \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") "
Nov 03 06:06:34.140911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.140884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\") pod \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\" (UID: \"9a70bce6-ee8b-4fd5-9400-d8f8944926d8\") "
Nov 03 06:06:34.141108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.141084    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca" pod "9a70bce6-ee8b-4fd5-9400-d8f8944926d8" (UID: "9a70bce6-ee8b-4fd5-9400-d8f8944926d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:34.141346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.141326    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:34.143088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.143056    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca" pod "9a70bce6-ee8b-4fd5-9400-d8f8944926d8" (UID: "9a70bce6-ee8b-4fd5-9400-d8f8944926d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:34.143088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.143058    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7" pod "9a70bce6-ee8b-4fd5-9400-d8f8944926d8" (UID: "9a70bce6-ee8b-4fd5-9400-d8f8944926d8"). InnerVolumeSpecName "kube-api-access-sfgd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:34.242418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.242326    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfgd7\" (UniqueName: \"kubernetes.io/projected/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-kube-api-access-sfgd7\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:34.242418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.242359    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a70bce6-ee8b-4fd5-9400-d8f8944926d8-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:34.550896 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.550818    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"]
Nov 03 06:06:34.551638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.551608    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9a70bce6-ee8b-4fd5-9400-d8f8944926d8" containerName="revoke-keys"
Nov 03 06:06:34.551638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.551633    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a70bce6-ee8b-4fd5-9400-d8f8944926d8" containerName="revoke-keys"
Nov 03 06:06:34.551856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.551808    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a70bce6-ee8b-4fd5-9400-d8f8944926d8" containerName="revoke-keys"
Nov 03 06:06:34.557296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.557266    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.573045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.573018    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"]
Nov 03 06:06:34.646947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.646906    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnlcz\" (UniqueName: \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.647130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.647038    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.647130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.647090    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.748494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.748463    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.748656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.748520    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.748728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.748652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnlcz\" (UniqueName: \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.749262 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.749239    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.750884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.750863    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.773678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.773648    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnlcz\" (UniqueName: \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\") pod \"maas-api-revoke-sub-4d1f8b40-pl44d\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.872047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.872010    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:34.961065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.961033    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"
Nov 03 06:06:34.961065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.961052    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b" event={"ID":"9a70bce6-ee8b-4fd5-9400-d8f8944926d8","Type":"ContainerDied","Data":"e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836"}
Nov 03 06:06:34.961509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.961103    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37bffbaddef523a5081a1a680a5c0811fa6f0a845352e7f70c9ef4dc7cd7836"
Nov 03 06:06:34.963185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.963155    4528 generic.go:356] "Generic (PLEG): container finished" podID="3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" containerID="4fea7f61c8e07859e07ba966f6a9a0438ce3066f687a1ffc6e20f9e10e78627e" exitCode=0
Nov 03 06:06:34.963332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:34.963232    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw" event={"ID":"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2","Type":"ContainerDied","Data":"4fea7f61c8e07859e07ba966f6a9a0438ce3066f687a1ffc6e20f9e10e78627e"}
Nov 03 06:06:35.119308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.119280    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:35.156998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.156911    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\") pod \"1402809e-9c66-445b-8d96-addd2cfed64f\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") "
Nov 03 06:06:35.157159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.157050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\") pod \"1402809e-9c66-445b-8d96-addd2cfed64f\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") "
Nov 03 06:06:35.157159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.157089    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\" (UniqueName: \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\") pod \"1402809e-9c66-445b-8d96-addd2cfed64f\" (UID: \"1402809e-9c66-445b-8d96-addd2cfed64f\") "
Nov 03 06:06:35.157608 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.157342    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca" pod "1402809e-9c66-445b-8d96-addd2cfed64f" (UID: "1402809e-9c66-445b-8d96-addd2cfed64f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:35.157727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.157711    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:35.159676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.159379    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca" pod "1402809e-9c66-445b-8d96-addd2cfed64f" (UID: "1402809e-9c66-445b-8d96-addd2cfed64f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:35.160200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.160151    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6" pod "1402809e-9c66-445b-8d96-addd2cfed64f" (UID: "1402809e-9c66-445b-8d96-addd2cfed64f"). InnerVolumeSpecName "kube-api-access-nmgc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:35.195628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.195598    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"]
Nov 03 06:06:35.196510 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:35.196485    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14494674_4433_4e53_b7bf_678bb16d5e70.slice/crio-e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363 WatchSource:0}: Error finding container e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363: Status 404 returned error can't find the container with id e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363
Nov 03 06:06:35.258755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.258719    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1402809e-9c66-445b-8d96-addd2cfed64f-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:35.258755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.258752    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmgc6\" (UniqueName: \"kubernetes.io/projected/1402809e-9c66-445b-8d96-addd2cfed64f-kube-api-access-nmgc6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:35.977685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.977632    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv" event={"ID":"1402809e-9c66-445b-8d96-addd2cfed64f","Type":"ContainerDied","Data":"0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9"}
Nov 03 06:06:35.978189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.977753    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0569cd1f77c0c376b38cec968b28f3beea42631acba2a1a37178426551eb8ae9"
Nov 03 06:06:35.978270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.978239    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"
Nov 03 06:06:35.980519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.980375    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d" event={"ID":"14494674-4433-4e53-b7bf-678bb16d5e70","Type":"ContainerStarted","Data":"f09449ee3a3fcc2cac0ef2b0c5915a69c61fc3e9dcd906b0c63ca320dfe46d2b"}
Nov 03 06:06:35.980519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:35.980446    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d" event={"ID":"14494674-4433-4e53-b7bf-678bb16d5e70","Type":"ContainerStarted","Data":"e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363"}
Nov 03 06:06:36.026006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.025895    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d" podStartSLOduration=2.025878646 podStartE2EDuration="2.025878646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:36.020524113 +0000 UTC m=+684.431921191" watchObservedRunningTime="2026-11-03 06:06:36.025878646 +0000 UTC m=+684.437275773"
Nov 03 06:06:36.185215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.185185    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:36.270302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.270264    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\" (UniqueName: \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\") pod \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") "
Nov 03 06:06:36.270451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.270395    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\") pod \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") "
Nov 03 06:06:36.270512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.270462    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\") pod \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\" (UID: \"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2\") "
Nov 03 06:06:36.270850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.270812    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca" pod "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" (UID: "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:36.272436 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.272410    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca" pod "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" (UID: "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:36.272531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.272463    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x" pod "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" (UID: "3d2d3b31-7a0e-40ae-a1b8-33b588b593d2"). InnerVolumeSpecName "kube-api-access-vdr9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:36.372325 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.372288    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:36.372325 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.372321    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:36.372325 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.372330    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdr9x\" (UniqueName: \"kubernetes.io/projected/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2-kube-api-access-vdr9x\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:36.991350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.991308    4528 generic.go:356] "Generic (PLEG): container finished" podID="14494674-4433-4e53-b7bf-678bb16d5e70" containerID="f09449ee3a3fcc2cac0ef2b0c5915a69c61fc3e9dcd906b0c63ca320dfe46d2b" exitCode=0
Nov 03 06:06:36.991856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.991415    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d" event={"ID":"14494674-4433-4e53-b7bf-678bb16d5e70","Type":"ContainerDied","Data":"f09449ee3a3fcc2cac0ef2b0c5915a69c61fc3e9dcd906b0c63ca320dfe46d2b"}
Nov 03 06:06:36.992934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.992903    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw" event={"ID":"3d2d3b31-7a0e-40ae-a1b8-33b588b593d2","Type":"ContainerDied","Data":"b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08"}
Nov 03 06:06:36.992934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.992930    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"
Nov 03 06:06:36.993131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:36.992956    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4053317597a90e455edce0048002c8a4bb2e622dd6163a00ad1d294ff7faa08"
Nov 03 06:06:38.162194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.162168    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:38.193180 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.193147    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\") pod \"14494674-4433-4e53-b7bf-678bb16d5e70\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") "
Nov 03 06:06:38.193341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.193237    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\") pod \"14494674-4433-4e53-b7bf-678bb16d5e70\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") "
Nov 03 06:06:38.193341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.193272    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\" (UniqueName: \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\") pod \"14494674-4433-4e53-b7bf-678bb16d5e70\" (UID: \"14494674-4433-4e53-b7bf-678bb16d5e70\") "
Nov 03 06:06:38.193565 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.193543    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca" pod "14494674-4433-4e53-b7bf-678bb16d5e70" (UID: "14494674-4433-4e53-b7bf-678bb16d5e70"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:38.193787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.193770    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:38.195906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.195867    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca" pod "14494674-4433-4e53-b7bf-678bb16d5e70" (UID: "14494674-4433-4e53-b7bf-678bb16d5e70"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:38.196009 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.195945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz" pod "14494674-4433-4e53-b7bf-678bb16d5e70" (UID: "14494674-4433-4e53-b7bf-678bb16d5e70"). InnerVolumeSpecName "kube-api-access-gnlcz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:38.295238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.295147    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14494674-4433-4e53-b7bf-678bb16d5e70-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:38.295238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:38.295185    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnlcz\" (UniqueName: \"kubernetes.io/projected/14494674-4433-4e53-b7bf-678bb16d5e70-kube-api-access-gnlcz\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:39.011755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:39.011724    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"
Nov 03 06:06:39.011955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:39.011768    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d" event={"ID":"14494674-4433-4e53-b7bf-678bb16d5e70","Type":"ContainerDied","Data":"e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363"}
Nov 03 06:06:39.011955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:39.011825    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e313a617c40f9e5d8ae1a0368559d4d72a3f1119340cf20ab3f9c727c3cf1363"
Nov 03 06:06:43.694186 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:06:43.694150    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:06:44.994327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994290    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"]
Nov 03 06:06:44.994862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994847    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" containerName="revoke-keys"
Nov 03 06:06:44.994921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994863    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" containerName="revoke-keys"
Nov 03 06:06:44.994921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994881    4528 cpu_manager.go:414] "containerMap: removing container" podUID="14494674-4433-4e53-b7bf-678bb16d5e70" containerName="revoke-keys"
Nov 03 06:06:44.994921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994887    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14494674-4433-4e53-b7bf-678bb16d5e70" containerName="revoke-keys"
Nov 03 06:06:44.994921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994908    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1402809e-9c66-445b-8d96-addd2cfed64f" containerName="revoke-keys"
Nov 03 06:06:44.994921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994914    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1402809e-9c66-445b-8d96-addd2cfed64f" containerName="revoke-keys"
Nov 03 06:06:44.995071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.994997    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="14494674-4433-4e53-b7bf-678bb16d5e70" containerName="revoke-keys"
Nov 03 06:06:44.995071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.995007    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1402809e-9c66-445b-8d96-addd2cfed64f" containerName="revoke-keys"
Nov 03 06:06:44.995071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.995020    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" containerName="revoke-keys"
Nov 03 06:06:44.998030 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:44.998010    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.017154 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.017126    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"]
Nov 03 06:06:45.023379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.023341    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:06:45.023554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.023358    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:06:45.023554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.023441    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:06:45.023748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.023462    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:06:45.098263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.098211    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njd7h\" (UniqueName: \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.098479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.098342    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.098479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.098459    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.200272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.200220    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njd7h\" (UniqueName: \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.200272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.200287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.200511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.200334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.201010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.200983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.202662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.202634    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.223000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.222970    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njd7h\" (UniqueName: \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\") pod \"maas-api-revoke-sub-4d10695c-7hg5f\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.315773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.315739    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:45.629090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:45.629050    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"]
Nov 03 06:06:45.629744 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:45.629705    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1a85e9e_f9be_4ec3_bcda_358cf1ecc22d.slice/crio-dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e WatchSource:0}: Error finding container dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e: Status 404 returned error can't find the container with id dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e
Nov 03 06:06:46.076979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:46.076941    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f" event={"ID":"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d","Type":"ContainerStarted","Data":"6a745d0c9d5ad90a4d85832d7ad8fc596f8a65152dcd2153269993e8b71cc0ae"}
Nov 03 06:06:46.077487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:46.077008    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f" event={"ID":"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d","Type":"ContainerStarted","Data":"dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e"}
Nov 03 06:06:46.121529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:46.121460    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f" podStartSLOduration=2.121441928 podStartE2EDuration="2.121441928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:46.11932592 +0000 UTC m=+694.530723011" watchObservedRunningTime="2026-11-03 06:06:46.121441928 +0000 UTC m=+694.532839007"
Nov 03 06:06:47.087607 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:47.087571    4528 generic.go:356] "Generic (PLEG): container finished" podID="f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" containerID="6a745d0c9d5ad90a4d85832d7ad8fc596f8a65152dcd2153269993e8b71cc0ae" exitCode=0
Nov 03 06:06:47.088032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:47.087644    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f" event={"ID":"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d","Type":"ContainerDied","Data":"6a745d0c9d5ad90a4d85832d7ad8fc596f8a65152dcd2153269993e8b71cc0ae"}
Nov 03 06:06:48.265205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.265178    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:48.439456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.439369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\") pod \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") "
Nov 03 06:06:48.439456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.439429    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\" (UniqueName: \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\") pod \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") "
Nov 03 06:06:48.439648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.439522    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\") pod \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\" (UID: \"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d\") "
Nov 03 06:06:48.439767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.439745    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca" pod "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" (UID: "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:48.440036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.440018    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:48.441560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.441536    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca" pod "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" (UID: "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:48.441639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.441597    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h" pod "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" (UID: "f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d"). InnerVolumeSpecName "kube-api-access-njd7h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:48.540924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.540888    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njd7h\" (UniqueName: \"kubernetes.io/projected/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-kube-api-access-njd7h\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:48.540924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:48.540919    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:49.103238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:49.103202    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f" event={"ID":"f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d","Type":"ContainerDied","Data":"dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e"}
Nov 03 06:06:49.103417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:49.103266    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd368ac4013cfbf8587e91eb1286034514fc2f89dbf320a8dd8da5722a971e2e"
Nov 03 06:06:49.103417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:49.103212    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"
Nov 03 06:06:51.356267 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.356224    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"]
Nov 03 06:06:51.356765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.356752    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" containerName="revoke-keys"
Nov 03 06:06:51.356816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.356766    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" containerName="revoke-keys"
Nov 03 06:06:51.356911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.356901    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" containerName="revoke-keys"
Nov 03 06:06:51.365177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.365151    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.379117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.379086    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"]
Nov 03 06:06:51.387990 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.387956    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:06:51.388860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.388838    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:06:51.389097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.389080    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:06:51.395926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.395903    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:06:51.464115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.464078    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"]
Nov 03 06:06:51.468575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.468549    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.475589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.475559    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q27bm\" (UniqueName: \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.475748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.475727    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.475816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.475781    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.493690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.493649    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"]
Nov 03 06:06:51.576696 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.576661    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl5x\" (UniqueName: \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.576872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.576761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q27bm\" (UniqueName: \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.576872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.576819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.576987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.576898    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.576987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.576922    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.577138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.577117    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.577565 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.577547    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.579603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.579581    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.600409 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.600378    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q27bm\" (UniqueName: \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\") pod \"maas-api-revoke-sub-bebf5711-n7jd5\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.678016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.677919    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.678197 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.678014    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl5x\" (UniqueName: \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.678197 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.678126    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.678756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.678725    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.680346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.680318    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.693448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.693421    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:51.701031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.701006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl5x\" (UniqueName: \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\") pod \"maas-api-revoke-sub-0e26fdc4-tzfdd\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.742093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.741636    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"]
Nov 03 06:06:51.749067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.749041    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.787320 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.777446    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"]
Nov 03 06:06:51.787320 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.784027    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:51.880862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.880819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.881079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.881061    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.881250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.881233    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnv99\" (UniqueName: \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.982450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.982363    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.982612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.982495    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.982612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.982597    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnv99\" (UniqueName: \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.983144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.983120    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:51.985108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:51.985082    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:52.005189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.005163    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnv99\" (UniqueName: \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\") pod \"maas-api-revoke-sub-1f828729-gz62x\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:52.011728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.011699    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"]
Nov 03 06:06:52.012286 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:52.012258    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9632817e_c0d0_4892_977c_8c4d8acbea77.slice/crio-4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1 WatchSource:0}: Error finding container 4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1: Status 404 returned error can't find the container with id 4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1
Nov 03 06:06:52.076901 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.076868    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:52.109011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.108939    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"]
Nov 03 06:06:52.110701 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:52.110674    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eead744_165b_4b91_94b9_8f389f100f11.slice/crio-09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028 WatchSource:0}: Error finding container 09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028: Status 404 returned error can't find the container with id 09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028
Nov 03 06:06:52.147671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.147637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5" event={"ID":"9632817e-c0d0-4892-977c-8c4d8acbea77","Type":"ContainerStarted","Data":"4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1"}
Nov 03 06:06:52.148867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.148833    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd" event={"ID":"5eead744-165b-4b91-94b9-8f389f100f11","Type":"ContainerStarted","Data":"09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028"}
Nov 03 06:06:52.407396 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:52.407365    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"]
Nov 03 06:06:52.407950 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:52.407922    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e0d557_008b_4ccc_8b83_9880d9b65ca2.slice/crio-b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086 WatchSource:0}: Error finding container b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086: Status 404 returned error can't find the container with id b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086
Nov 03 06:06:53.157400 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.157361    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x" event={"ID":"40e0d557-008b-4ccc-8b83-9880d9b65ca2","Type":"ContainerStarted","Data":"d17c77135d68d3ebf630574edffec6c273688fdcd6b6aa23c970a8bb9e44d0bd"}
Nov 03 06:06:53.157599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.157425    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x" event={"ID":"40e0d557-008b-4ccc-8b83-9880d9b65ca2","Type":"ContainerStarted","Data":"b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086"}
Nov 03 06:06:53.159001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.158966    4528 generic.go:356] "Generic (PLEG): container finished" podID="9632817e-c0d0-4892-977c-8c4d8acbea77" containerID="ec9ca0ab412323025cb5fc55f7efb5b8a5758913926d7874036ed92a3986389b" exitCode=0
Nov 03 06:06:53.159126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.159044    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5" event={"ID":"9632817e-c0d0-4892-977c-8c4d8acbea77","Type":"ContainerDied","Data":"ec9ca0ab412323025cb5fc55f7efb5b8a5758913926d7874036ed92a3986389b"}
Nov 03 06:06:53.160463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.160444    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd" event={"ID":"5eead744-165b-4b91-94b9-8f389f100f11","Type":"ContainerStarted","Data":"a114b5cdcbbc77e80c85328c2f19d0f5757f59a25485413fafce0e29ea8fe632"}
Nov 03 06:06:53.197818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.197752    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x" podStartSLOduration=2.197737492 podStartE2EDuration="2.197737492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:53.195985825 +0000 UTC m=+701.607382917" watchObservedRunningTime="2026-11-03 06:06:53.197737492 +0000 UTC m=+701.609134564"
Nov 03 06:06:53.241825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.241751    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd" podStartSLOduration=2.241737455 podStartE2EDuration="2.241737455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:53.238521943 +0000 UTC m=+701.649919016" watchObservedRunningTime="2026-11-03 06:06:53.241737455 +0000 UTC m=+701.653134565"
Nov 03 06:06:53.850385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.850346    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"]
Nov 03 06:06:53.854699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.854678    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:53.872127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:53.872099    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"]
Nov 03 06:06:54.011298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.011254    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.011495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.011376    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhks4\" (UniqueName: \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.011495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.011430    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.112602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.112518    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.112760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.112607    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhks4\" (UniqueName: \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.112760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.112630    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.113333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.113306    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.115103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.115078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.135340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.135296    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhks4\" (UniqueName: \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\") pod \"maas-api-revoke-sub-79ab5ecb-8wjts\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.169974 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.169940    4528 generic.go:356] "Generic (PLEG): container finished" podID="40e0d557-008b-4ccc-8b83-9880d9b65ca2" containerID="d17c77135d68d3ebf630574edffec6c273688fdcd6b6aa23c970a8bb9e44d0bd" exitCode=0
Nov 03 06:06:54.170164 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.170026    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x" event={"ID":"40e0d557-008b-4ccc-8b83-9880d9b65ca2","Type":"ContainerDied","Data":"d17c77135d68d3ebf630574edffec6c273688fdcd6b6aa23c970a8bb9e44d0bd"}
Nov 03 06:06:54.170222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.170182    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:54.171861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.171839    4528 generic.go:356] "Generic (PLEG): container finished" podID="5eead744-165b-4b91-94b9-8f389f100f11" containerID="a114b5cdcbbc77e80c85328c2f19d0f5757f59a25485413fafce0e29ea8fe632" exitCode=0
Nov 03 06:06:54.171954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.171895    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd" event={"ID":"5eead744-165b-4b91-94b9-8f389f100f11","Type":"ContainerDied","Data":"a114b5cdcbbc77e80c85328c2f19d0f5757f59a25485413fafce0e29ea8fe632"}
Nov 03 06:06:54.376232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.376166    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:54.415491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.415448    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\" (UniqueName: \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\") pod \"9632817e-c0d0-4892-977c-8c4d8acbea77\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") "
Nov 03 06:06:54.415744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.415552    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\") pod \"9632817e-c0d0-4892-977c-8c4d8acbea77\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") "
Nov 03 06:06:54.415744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.415591    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\") pod \"9632817e-c0d0-4892-977c-8c4d8acbea77\" (UID: \"9632817e-c0d0-4892-977c-8c4d8acbea77\") "
Nov 03 06:06:54.416554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.416523    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca" pod "9632817e-c0d0-4892-977c-8c4d8acbea77" (UID: "9632817e-c0d0-4892-977c-8c4d8acbea77"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:54.421493 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.421459    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm" pod "9632817e-c0d0-4892-977c-8c4d8acbea77" (UID: "9632817e-c0d0-4892-977c-8c4d8acbea77"). InnerVolumeSpecName "kube-api-access-q27bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:54.422327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.422193    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca" pod "9632817e-c0d0-4892-977c-8c4d8acbea77" (UID: "9632817e-c0d0-4892-977c-8c4d8acbea77"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:54.485665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.485553    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"]
Nov 03 06:06:54.488440 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:54.488407    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd847f_72bd_4a71_87af_1032f3dd0313.slice/crio-c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28 WatchSource:0}: Error finding container c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28: Status 404 returned error can't find the container with id c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28
Nov 03 06:06:54.516661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.516633    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q27bm\" (UniqueName: \"kubernetes.io/projected/9632817e-c0d0-4892-977c-8c4d8acbea77-kube-api-access-q27bm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:54.516661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.516659    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:54.516832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:54.516671    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9632817e-c0d0-4892-977c-8c4d8acbea77-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.180622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.180579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts" event={"ID":"e5dd847f-72bd-4a71-87af-1032f3dd0313","Type":"ContainerStarted","Data":"0ae8b18c1f658167e36624e99f86b62c01a46632513e968606d174573472cef6"}
Nov 03 06:06:55.181446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.181414    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts" event={"ID":"e5dd847f-72bd-4a71-87af-1032f3dd0313","Type":"ContainerStarted","Data":"c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28"}
Nov 03 06:06:55.182576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.182549    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"
Nov 03 06:06:55.182708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.182575    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5" event={"ID":"9632817e-c0d0-4892-977c-8c4d8acbea77","Type":"ContainerDied","Data":"4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1"}
Nov 03 06:06:55.182708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.182618    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b633c344588888eeac22c9bba5b07a2bbf8fe8beb026f8b02cf3c2221a4fbd1"
Nov 03 06:06:55.233642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.228556    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts" podStartSLOduration=2.228536404 podStartE2EDuration="2.228536404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:06:55.21961127 +0000 UTC m=+703.631008350" watchObservedRunningTime="2026-11-03 06:06:55.228536404 +0000 UTC m=+703.639933483"
Nov 03 06:06:55.475286 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.475260    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:55.485846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.485818    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:55.526983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.526949    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\") pod \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") "
Nov 03 06:06:55.527233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527055    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\" (UniqueName: \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\") pod \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") "
Nov 03 06:06:55.527233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527093    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\") pod \"5eead744-165b-4b91-94b9-8f389f100f11\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") "
Nov 03 06:06:55.527233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527173    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\") pod \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\" (UID: \"40e0d557-008b-4ccc-8b83-9880d9b65ca2\") "
Nov 03 06:06:55.527233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527205    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\" (UniqueName: \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\") pod \"5eead744-165b-4b91-94b9-8f389f100f11\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") "
Nov 03 06:06:55.527233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527233    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\") pod \"5eead744-165b-4b91-94b9-8f389f100f11\" (UID: \"5eead744-165b-4b91-94b9-8f389f100f11\") "
Nov 03 06:06:55.527616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527581    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca" pod "5eead744-165b-4b91-94b9-8f389f100f11" (UID: "5eead744-165b-4b91-94b9-8f389f100f11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:55.527921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527896    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.527995 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.527927    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca" pod "40e0d557-008b-4ccc-8b83-9880d9b65ca2" (UID: "40e0d557-008b-4ccc-8b83-9880d9b65ca2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:55.532818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.529808    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca" pod "40e0d557-008b-4ccc-8b83-9880d9b65ca2" (UID: "40e0d557-008b-4ccc-8b83-9880d9b65ca2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:55.532818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.530038    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x" pod "5eead744-165b-4b91-94b9-8f389f100f11" (UID: "5eead744-165b-4b91-94b9-8f389f100f11"). InnerVolumeSpecName "kube-api-access-lcl5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:55.532818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.530055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca" pod "5eead744-165b-4b91-94b9-8f389f100f11" (UID: "5eead744-165b-4b91-94b9-8f389f100f11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:55.532818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.530426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99" pod "40e0d557-008b-4ccc-8b83-9880d9b65ca2" (UID: "40e0d557-008b-4ccc-8b83-9880d9b65ca2"). InnerVolumeSpecName "kube-api-access-nnv99". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:55.628586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.628554    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.628586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.628581    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcl5x\" (UniqueName: \"kubernetes.io/projected/5eead744-165b-4b91-94b9-8f389f100f11-kube-api-access-lcl5x\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.628586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.628591    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5eead744-165b-4b91-94b9-8f389f100f11-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.628586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.628599    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40e0d557-008b-4ccc-8b83-9880d9b65ca2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:55.628586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:55.628608    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnv99\" (UniqueName: \"kubernetes.io/projected/40e0d557-008b-4ccc-8b83-9880d9b65ca2-kube-api-access-nnv99\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:56.190913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.190877    4528 generic.go:356] "Generic (PLEG): container finished" podID="e5dd847f-72bd-4a71-87af-1032f3dd0313" containerID="0ae8b18c1f658167e36624e99f86b62c01a46632513e968606d174573472cef6" exitCode=0
Nov 03 06:06:56.191391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.190963    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts" event={"ID":"e5dd847f-72bd-4a71-87af-1032f3dd0313","Type":"ContainerDied","Data":"0ae8b18c1f658167e36624e99f86b62c01a46632513e968606d174573472cef6"}
Nov 03 06:06:56.197041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.197011    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"
Nov 03 06:06:56.197041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.197028    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x" event={"ID":"40e0d557-008b-4ccc-8b83-9880d9b65ca2","Type":"ContainerDied","Data":"b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086"}
Nov 03 06:06:56.197284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.197087    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1af6071fb77bc60a6f6bb52b20aae00990230d318ce0df56605cddc2836f086"
Nov 03 06:06:56.198984 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.198907    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"
Nov 03 06:06:56.198984 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.198952    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd" event={"ID":"5eead744-165b-4b91-94b9-8f389f100f11","Type":"ContainerDied","Data":"09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028"}
Nov 03 06:06:56.199143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.198998    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="09a01a42fab2945302838a998cf047cbd168a010680b7e83a208da6371494028"
Nov 03 06:06:56.572239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572204    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"]
Nov 03 06:06:56.572804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572781    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9632817e-c0d0-4892-977c-8c4d8acbea77" containerName="revoke-keys"
Nov 03 06:06:56.572863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572806    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9632817e-c0d0-4892-977c-8c4d8acbea77" containerName="revoke-keys"
Nov 03 06:06:56.572863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572824    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5eead744-165b-4b91-94b9-8f389f100f11" containerName="revoke-keys"
Nov 03 06:06:56.572863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572829    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5eead744-165b-4b91-94b9-8f389f100f11" containerName="revoke-keys"
Nov 03 06:06:56.572863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572837    4528 cpu_manager.go:414] "containerMap: removing container" podUID="40e0d557-008b-4ccc-8b83-9880d9b65ca2" containerName="revoke-keys"
Nov 03 06:06:56.572863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572842    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e0d557-008b-4ccc-8b83-9880d9b65ca2" containerName="revoke-keys"
Nov 03 06:06:56.573010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572945    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5eead744-165b-4b91-94b9-8f389f100f11" containerName="revoke-keys"
Nov 03 06:06:56.573010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572954    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e0d557-008b-4ccc-8b83-9880d9b65ca2" containerName="revoke-keys"
Nov 03 06:06:56.573010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.572965    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9632817e-c0d0-4892-977c-8c4d8acbea77" containerName="revoke-keys"
Nov 03 06:06:56.576587 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.576570    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.597610 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.597585    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"]
Nov 03 06:06:56.640501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.640464    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.640683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.640565    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.640683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.640626    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptmtj\" (UniqueName: \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.742105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.742056    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptmtj\" (UniqueName: \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.742316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.742186    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.742316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.742234    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.742873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.742849    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.744570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.744550    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.764560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.764531    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptmtj\" (UniqueName: \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\") pod \"maas-api-revoke-sub-2243a7ae-rzvgn\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:56.890620 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:56.890536    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:57.205233 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:06:57.205204    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed290cc_5cf2_4c52_a083_b96d9cee9cd2.slice/crio-cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc WatchSource:0}: Error finding container cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc: Status 404 returned error can't find the container with id cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc
Nov 03 06:06:57.206407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.206375    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"]
Nov 03 06:06:57.425282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.425257    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:57.451281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.451248    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\") pod \"e5dd847f-72bd-4a71-87af-1032f3dd0313\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") "
Nov 03 06:06:57.451427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.451370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\" (UniqueName: \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\") pod \"e5dd847f-72bd-4a71-87af-1032f3dd0313\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") "
Nov 03 06:06:57.451511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.451495    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\") pod \"e5dd847f-72bd-4a71-87af-1032f3dd0313\" (UID: \"e5dd847f-72bd-4a71-87af-1032f3dd0313\") "
Nov 03 06:06:57.451994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.451918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca" pod "e5dd847f-72bd-4a71-87af-1032f3dd0313" (UID: "e5dd847f-72bd-4a71-87af-1032f3dd0313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:57.453430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.453403    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca" pod "e5dd847f-72bd-4a71-87af-1032f3dd0313" (UID: "e5dd847f-72bd-4a71-87af-1032f3dd0313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:57.453530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.453502    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4" pod "e5dd847f-72bd-4a71-87af-1032f3dd0313" (UID: "e5dd847f-72bd-4a71-87af-1032f3dd0313"). InnerVolumeSpecName "kube-api-access-zhks4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:57.553042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.553006    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhks4\" (UniqueName: \"kubernetes.io/projected/e5dd847f-72bd-4a71-87af-1032f3dd0313-kube-api-access-zhks4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:57.553042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.553036    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:57.553042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:57.553048    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e5dd847f-72bd-4a71-87af-1032f3dd0313-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:58.216528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.216494    4528 generic.go:356] "Generic (PLEG): container finished" podID="eed290cc-5cf2-4c52-a083-b96d9cee9cd2" containerID="42c76f5ef799bae37e8fb95b0e37c1fae636fb973c30a618cc9b9e2e33f8c2c1" exitCode=0
Nov 03 06:06:58.216982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.216579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn" event={"ID":"eed290cc-5cf2-4c52-a083-b96d9cee9cd2","Type":"ContainerDied","Data":"42c76f5ef799bae37e8fb95b0e37c1fae636fb973c30a618cc9b9e2e33f8c2c1"}
Nov 03 06:06:58.216982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.216657    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn" event={"ID":"eed290cc-5cf2-4c52-a083-b96d9cee9cd2","Type":"ContainerStarted","Data":"cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc"}
Nov 03 06:06:58.218210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.218192    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"
Nov 03 06:06:58.218344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.218219    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts" event={"ID":"e5dd847f-72bd-4a71-87af-1032f3dd0313","Type":"ContainerDied","Data":"c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28"}
Nov 03 06:06:58.218344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:58.218265    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9a576d744d9507d91389fa3b5798f01542bf54ab7f1fc7dd58de6d97f078e28"
Nov 03 06:06:59.415035 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.414999    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:06:59.494152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.494055    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\") pod \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") "
Nov 03 06:06:59.494152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.494114    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\" (UniqueName: \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\") pod \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") "
Nov 03 06:06:59.494334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.494185    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\") pod \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\" (UID: \"eed290cc-5cf2-4c52-a083-b96d9cee9cd2\") "
Nov 03 06:06:59.494810 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.494756    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca" pod "eed290cc-5cf2-4c52-a083-b96d9cee9cd2" (UID: "eed290cc-5cf2-4c52-a083-b96d9cee9cd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:06:59.496734 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.496709    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca" pod "eed290cc-5cf2-4c52-a083-b96d9cee9cd2" (UID: "eed290cc-5cf2-4c52-a083-b96d9cee9cd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:06:59.496870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.496842    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj" pod "eed290cc-5cf2-4c52-a083-b96d9cee9cd2" (UID: "eed290cc-5cf2-4c52-a083-b96d9cee9cd2"). InnerVolumeSpecName "kube-api-access-ptmtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:06:59.596044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.596002    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:59.596044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.596036    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptmtj\" (UniqueName: \"kubernetes.io/projected/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-kube-api-access-ptmtj\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:06:59.596044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:06:59.596050    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eed290cc-5cf2-4c52-a083-b96d9cee9cd2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:00.235872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:00.235841    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"
Nov 03 06:07:00.235872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:00.235851    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn" event={"ID":"eed290cc-5cf2-4c52-a083-b96d9cee9cd2","Type":"ContainerDied","Data":"cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc"}
Nov 03 06:07:00.236083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:00.235903    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf560c17133b66ce64c8fe5222c30eda9b251746fe164337fb0164d10555babc"
Nov 03 06:07:04.438437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.437812    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"]
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438733    4528 cpu_manager.go:414] "containerMap: removing container" podUID="eed290cc-5cf2-4c52-a083-b96d9cee9cd2" containerName="revoke-keys"
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438755    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed290cc-5cf2-4c52-a083-b96d9cee9cd2" containerName="revoke-keys"
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438810    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e5dd847f-72bd-4a71-87af-1032f3dd0313" containerName="revoke-keys"
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438820    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5dd847f-72bd-4a71-87af-1032f3dd0313" containerName="revoke-keys"
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438960    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5dd847f-72bd-4a71-87af-1032f3dd0313" containerName="revoke-keys"
Nov 03 06:07:04.439048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.438980    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed290cc-5cf2-4c52-a083-b96d9cee9cd2" containerName="revoke-keys"
Nov 03 06:07:04.443519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.443492    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.458066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.458034    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"]
Nov 03 06:07:04.462880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.462854    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:07:04.462880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.462876    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:07:04.463202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.463176    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:07:04.463302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.463278    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:07:04.620937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.607093    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.620937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.607297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs8zb\" (UniqueName: \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.620937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.607389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.723865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.723082    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.723865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.723182    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs8zb\" (UniqueName: \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.723865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.723216    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.747760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.734288    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.747760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.745980    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.758105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.758079    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs8zb\" (UniqueName: \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\") pod \"maas-api-revoke-sub-f754b51d-nrpvk\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:04.762925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:04.762897    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:05.214362 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:05.204248    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"]
Nov 03 06:07:05.349772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:05.349705    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk" event={"ID":"d8cdb804-9451-4a70-ad7f-224e18470bc9","Type":"ContainerStarted","Data":"f739632bc7dff73af423e1598b77c643c9ac15cc6bb12e10a17f3d84f3b7b242"}
Nov 03 06:07:06.254378 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.254335    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"]
Nov 03 06:07:06.258600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.258577    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.265785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.265752    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.265934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.265804    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv9hr\" (UniqueName: \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.265975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.265951    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.276371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.276339    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"]
Nov 03 06:07:06.359304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.359265    4528 generic.go:356] "Generic (PLEG): container finished" podID="d8cdb804-9451-4a70-ad7f-224e18470bc9" containerID="6b74b2cb3c1fd5c2b46b34c6d905b5e9fcd3160aae19488d0ce78cea8a1fc997" exitCode=0
Nov 03 06:07:06.359479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.359330    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk" event={"ID":"d8cdb804-9451-4a70-ad7f-224e18470bc9","Type":"ContainerDied","Data":"6b74b2cb3c1fd5c2b46b34c6d905b5e9fcd3160aae19488d0ce78cea8a1fc997"}
Nov 03 06:07:06.366966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.366943    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.367088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.367073    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.367128 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.367097    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv9hr\" (UniqueName: \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.367714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.367690    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.369455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.369431    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.389600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.389568    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv9hr\" (UniqueName: \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\") pod \"maas-api-revoke-sub-8510b0c3-l9zh4\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.480232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.480198    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"]
Nov 03 06:07:06.484542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.484522    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.511097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.511007    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"]
Nov 03 06:07:06.569556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.569516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.569749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.569685    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.569850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.569827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsfrg\" (UniqueName: \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.574280 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.574259    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:06.673827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.671069    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsfrg\" (UniqueName: \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.673827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.671212    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.673827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.671360    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.673827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.672210    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.674988 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.674964    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.698382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.698346    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsfrg\" (UniqueName: \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\") pod \"maas-api-revoke-sub-5d26ce03-2mr6m\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.800232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.800124    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:06.908052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:06.908021    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"]
Nov 03 06:07:06.915841 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:06.915787    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod743e4996_6f32_412a_9d75_b0bab9479c7e.slice/crio-7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9 WatchSource:0}: Error finding container 7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9: Status 404 returned error can't find the container with id 7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9
Nov 03 06:07:07.103938 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:07.103904    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07e8ca0e_7e80_4205_8a69_3b679fce4c3d.slice/crio-ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544 WatchSource:0}: Error finding container ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544: Status 404 returned error can't find the container with id ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544
Nov 03 06:07:07.106095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.106063    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"]
Nov 03 06:07:07.369439 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.369337    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4" event={"ID":"743e4996-6f32-412a-9d75-b0bab9479c7e","Type":"ContainerStarted","Data":"3ca58c491e59ccb4221967bc4a0d9f553a720a3ade6ac288c32b960cc05f16fe"}
Nov 03 06:07:07.369439 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.369394    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4" event={"ID":"743e4996-6f32-412a-9d75-b0bab9479c7e","Type":"ContainerStarted","Data":"7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9"}
Nov 03 06:07:07.370825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.370769    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m" event={"ID":"07e8ca0e-7e80-4205-8a69-3b679fce4c3d","Type":"ContainerStarted","Data":"ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544"}
Nov 03 06:07:07.409647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.409593    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4" podStartSLOduration=1.409575692 podStartE2EDuration="1.409575692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:07.409431003 +0000 UTC m=+715.820828081" watchObservedRunningTime="2026-11-03 06:07:07.409575692 +0000 UTC m=+715.820972773"
Nov 03 06:07:07.520575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.520547    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:07.583091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.583057    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\") pod \"d8cdb804-9451-4a70-ad7f-224e18470bc9\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") "
Nov 03 06:07:07.583285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.583178    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\") pod \"d8cdb804-9451-4a70-ad7f-224e18470bc9\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") "
Nov 03 06:07:07.583285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.583250    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\" (UniqueName: \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\") pod \"d8cdb804-9451-4a70-ad7f-224e18470bc9\" (UID: \"d8cdb804-9451-4a70-ad7f-224e18470bc9\") "
Nov 03 06:07:07.583560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.583526    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca" pod "d8cdb804-9451-4a70-ad7f-224e18470bc9" (UID: "d8cdb804-9451-4a70-ad7f-224e18470bc9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:07.583763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.583736    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:07.585445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.585420    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca" pod "d8cdb804-9451-4a70-ad7f-224e18470bc9" (UID: "d8cdb804-9451-4a70-ad7f-224e18470bc9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:07.585561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.585454    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb" pod "d8cdb804-9451-4a70-ad7f-224e18470bc9" (UID: "d8cdb804-9451-4a70-ad7f-224e18470bc9"). InnerVolumeSpecName "kube-api-access-zs8zb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:07.685486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.685400    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8cdb804-9451-4a70-ad7f-224e18470bc9-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:07.685486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:07.685432    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs8zb\" (UniqueName: \"kubernetes.io/projected/d8cdb804-9451-4a70-ad7f-224e18470bc9-kube-api-access-zs8zb\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:08.070645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.070608    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"]
Nov 03 06:07:08.071254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.071238    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d8cdb804-9451-4a70-ad7f-224e18470bc9" containerName="revoke-keys"
Nov 03 06:07:08.071307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.071256    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8cdb804-9451-4a70-ad7f-224e18470bc9" containerName="revoke-keys"
Nov 03 06:07:08.071408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.071397    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8cdb804-9451-4a70-ad7f-224e18470bc9" containerName="revoke-keys"
Nov 03 06:07:08.074949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.074929    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.090269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.090216    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.090450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.090306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.090450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.090338    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mpf4\" (UniqueName: \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.095868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.095834    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"]
Nov 03 06:07:08.191978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.191946    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.192154 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.191998    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.192154 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.192022    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mpf4\" (UniqueName: \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.192758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.192731    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.194257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.194235    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.215226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.215191    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mpf4\" (UniqueName: \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\") pod \"maas-api-revoke-sub-8180f860-486t8\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.389537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.389499    4528 generic.go:356] "Generic (PLEG): container finished" podID="743e4996-6f32-412a-9d75-b0bab9479c7e" containerID="3ca58c491e59ccb4221967bc4a0d9f553a720a3ade6ac288c32b960cc05f16fe" exitCode=0
Nov 03 06:07:08.389997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.389579    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4" event={"ID":"743e4996-6f32-412a-9d75-b0bab9479c7e","Type":"ContainerDied","Data":"3ca58c491e59ccb4221967bc4a0d9f553a720a3ade6ac288c32b960cc05f16fe"}
Nov 03 06:07:08.391355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.391330    4528 generic.go:356] "Generic (PLEG): container finished" podID="07e8ca0e-7e80-4205-8a69-3b679fce4c3d" containerID="e80f9ce3b041a37fdcf7f008586df23393670c1f67396234403334fa32645a08" exitCode=0
Nov 03 06:07:08.391471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.391414    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m" event={"ID":"07e8ca0e-7e80-4205-8a69-3b679fce4c3d","Type":"ContainerDied","Data":"e80f9ce3b041a37fdcf7f008586df23393670c1f67396234403334fa32645a08"}
Nov 03 06:07:08.392976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.392950    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk" event={"ID":"d8cdb804-9451-4a70-ad7f-224e18470bc9","Type":"ContainerDied","Data":"f739632bc7dff73af423e1598b77c643c9ac15cc6bb12e10a17f3d84f3b7b242"}
Nov 03 06:07:08.392976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.392968    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"
Nov 03 06:07:08.393100 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.392987    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f739632bc7dff73af423e1598b77c643c9ac15cc6bb12e10a17f3d84f3b7b242"
Nov 03 06:07:08.444233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.444198    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:08.754874 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:08.754777    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e66745e_7b47_4051_b859_d04978db963a.slice/crio-74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce WatchSource:0}: Error finding container 74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce: Status 404 returned error can't find the container with id 74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce
Nov 03 06:07:08.755179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.755154    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"]
Nov 03 06:07:08.902065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.902030    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"]
Nov 03 06:07:08.907610 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.907586    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:08.924527 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:08.924490    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"]
Nov 03 06:07:09.000310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.000270    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sjm4\" (UniqueName: \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.000310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.000314    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.000580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.000467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.102136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.102102    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sjm4\" (UniqueName: \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.102136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.102141    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.102365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.102242    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.102971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.102944    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.104490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.104472    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.124913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.124872    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sjm4\" (UniqueName: \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\") pod \"maas-api-revoke-sub-bb043c0a-9mvdj\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.223712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.223671    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:09.402114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.402017    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8" event={"ID":"9e66745e-7b47-4051-b859-d04978db963a","Type":"ContainerStarted","Data":"0b27d20c396a85871254611f379212a172721a6dd258a1416454973c1d359130"}
Nov 03 06:07:09.402114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.402076    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8" event={"ID":"9e66745e-7b47-4051-b859-d04978db963a","Type":"ContainerStarted","Data":"74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce"}
Nov 03 06:07:09.446235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.446160    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8" podStartSLOduration=1.446139343 podStartE2EDuration="1.446139343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:09.442653147 +0000 UTC m=+717.854050226" watchObservedRunningTime="2026-11-03 06:07:09.446139343 +0000 UTC m=+717.857536426"
Nov 03 06:07:09.535369 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:09.535303    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b0e5da2_a514_4c3d_961e_bb9a2d8c1d08.slice/crio-2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218 WatchSource:0}: Error finding container 2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218: Status 404 returned error can't find the container with id 2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218
Nov 03 06:07:09.543062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.542984    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"]
Nov 03 06:07:09.563486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.563433    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:09.599566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.599539    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:09.608300 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.608119    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\") pod \"743e4996-6f32-412a-9d75-b0bab9479c7e\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") "
Nov 03 06:07:09.608486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.608376    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\" (UniqueName: \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\") pod \"743e4996-6f32-412a-9d75-b0bab9479c7e\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") "
Nov 03 06:07:09.608486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.608474    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\") pod \"743e4996-6f32-412a-9d75-b0bab9479c7e\" (UID: \"743e4996-6f32-412a-9d75-b0bab9479c7e\") "
Nov 03 06:07:09.608976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.608945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca" pod "743e4996-6f32-412a-9d75-b0bab9479c7e" (UID: "743e4996-6f32-412a-9d75-b0bab9479c7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:09.610931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.610886    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr" pod "743e4996-6f32-412a-9d75-b0bab9479c7e" (UID: "743e4996-6f32-412a-9d75-b0bab9479c7e"). InnerVolumeSpecName "kube-api-access-mv9hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:09.612843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.611109    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca" pod "743e4996-6f32-412a-9d75-b0bab9479c7e" (UID: "743e4996-6f32-412a-9d75-b0bab9479c7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:09.710049 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710009    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\") pod \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") "
Nov 03 06:07:09.710242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710169    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\" (UniqueName: \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\") pod \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") "
Nov 03 06:07:09.710242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710206    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\") pod \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\" (UID: \"07e8ca0e-7e80-4205-8a69-3b679fce4c3d\") "
Nov 03 06:07:09.710466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710435    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca" pod "07e8ca0e-7e80-4205-8a69-3b679fce4c3d" (UID: "07e8ca0e-7e80-4205-8a69-3b679fce4c3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:09.710674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710649    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv9hr\" (UniqueName: \"kubernetes.io/projected/743e4996-6f32-412a-9d75-b0bab9479c7e-kube-api-access-mv9hr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:09.710674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710675    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:09.710853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710689    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:09.710853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.710702    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/743e4996-6f32-412a-9d75-b0bab9479c7e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:09.779283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.779235    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca" pod "07e8ca0e-7e80-4205-8a69-3b679fce4c3d" (UID: "07e8ca0e-7e80-4205-8a69-3b679fce4c3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:09.779424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.779307    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg" pod "07e8ca0e-7e80-4205-8a69-3b679fce4c3d" (UID: "07e8ca0e-7e80-4205-8a69-3b679fce4c3d"). InnerVolumeSpecName "kube-api-access-fsfrg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:09.812090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.812054    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsfrg\" (UniqueName: \"kubernetes.io/projected/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-kube-api-access-fsfrg\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:09.812320 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:09.812297    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07e8ca0e-7e80-4205-8a69-3b679fce4c3d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:10.410546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.410518    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"
Nov 03 06:07:10.410978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.410516    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m" event={"ID":"07e8ca0e-7e80-4205-8a69-3b679fce4c3d","Type":"ContainerDied","Data":"ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544"}
Nov 03 06:07:10.410978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.410650    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae938f173a02b8a8681410a4b50739e7786ee16e6f3ef18a211b586473408544"
Nov 03 06:07:10.412319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.412291    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj" event={"ID":"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08","Type":"ContainerStarted","Data":"4d58850bdbe1a0aa05f3d0c6d0cd97182de9476dfd71104e61611211a4a6537a"}
Nov 03 06:07:10.412416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.412345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj" event={"ID":"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08","Type":"ContainerStarted","Data":"2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218"}
Nov 03 06:07:10.414172 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.414147    4528 generic.go:356] "Generic (PLEG): container finished" podID="9e66745e-7b47-4051-b859-d04978db963a" containerID="0b27d20c396a85871254611f379212a172721a6dd258a1416454973c1d359130" exitCode=0
Nov 03 06:07:10.414260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.414233    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8" event={"ID":"9e66745e-7b47-4051-b859-d04978db963a","Type":"ContainerDied","Data":"0b27d20c396a85871254611f379212a172721a6dd258a1416454973c1d359130"}
Nov 03 06:07:10.415637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.415615    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4" event={"ID":"743e4996-6f32-412a-9d75-b0bab9479c7e","Type":"ContainerDied","Data":"7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9"}
Nov 03 06:07:10.415724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.415650    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"
Nov 03 06:07:10.415724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.415655    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c3b325be0484e8c041276bb55ef92bc8119d59e43aa1bcf7bcf41e6728056d9"
Nov 03 06:07:10.460309 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:10.460248    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj" podStartSLOduration=2.460234017 podStartE2EDuration="2.460234017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:10.453483372 +0000 UTC m=+718.864880449" watchObservedRunningTime="2026-11-03 06:07:10.460234017 +0000 UTC m=+718.871631095"
Nov 03 06:07:11.424974 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.424939    4528 generic.go:356] "Generic (PLEG): container finished" podID="7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" containerID="4d58850bdbe1a0aa05f3d0c6d0cd97182de9476dfd71104e61611211a4a6537a" exitCode=0
Nov 03 06:07:11.425382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.425028    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj" event={"ID":"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08","Type":"ContainerDied","Data":"4d58850bdbe1a0aa05f3d0c6d0cd97182de9476dfd71104e61611211a4a6537a"}
Nov 03 06:07:11.591110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.591076    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:11.633390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.633353    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\") pod \"9e66745e-7b47-4051-b859-d04978db963a\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") "
Nov 03 06:07:11.633578 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.633492    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\" (UniqueName: \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\") pod \"9e66745e-7b47-4051-b859-d04978db963a\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") "
Nov 03 06:07:11.633654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.633636    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\") pod \"9e66745e-7b47-4051-b859-d04978db963a\" (UID: \"9e66745e-7b47-4051-b859-d04978db963a\") "
Nov 03 06:07:11.633781 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.633753    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca" pod "9e66745e-7b47-4051-b859-d04978db963a" (UID: "9e66745e-7b47-4051-b859-d04978db963a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:11.634504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.634163    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e66745e-7b47-4051-b859-d04978db963a-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:11.640935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.640886    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca" pod "9e66745e-7b47-4051-b859-d04978db963a" (UID: "9e66745e-7b47-4051-b859-d04978db963a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:11.641052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.640928    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4" pod "9e66745e-7b47-4051-b859-d04978db963a" (UID: "9e66745e-7b47-4051-b859-d04978db963a"). InnerVolumeSpecName "kube-api-access-5mpf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:11.735563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.735480    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mpf4\" (UniqueName: \"kubernetes.io/projected/9e66745e-7b47-4051-b859-d04978db963a-kube-api-access-5mpf4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:11.735563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:11.735512    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e66745e-7b47-4051-b859-d04978db963a-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:12.434535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.434498    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8" event={"ID":"9e66745e-7b47-4051-b859-d04978db963a","Type":"ContainerDied","Data":"74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce"}
Nov 03 06:07:12.434535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.434520    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"
Nov 03 06:07:12.435040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.434554    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74e39cee45566f03637575c4358b5c461d30b5d99d96267fe7c2e774549fd4ce"
Nov 03 06:07:12.586707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.586681    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:12.646179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.646138    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\") pod \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") "
Nov 03 06:07:12.646350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.646269    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\") pod \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") "
Nov 03 06:07:12.646420 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.646404    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\" (UniqueName: \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\") pod \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\" (UID: \"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08\") "
Nov 03 06:07:12.646643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.646613    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca" pod "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" (UID: "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:12.646956 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.646935    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:12.648930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.648904    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4" pod "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" (UID: "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08"). InnerVolumeSpecName "kube-api-access-9sjm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:12.649021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.648998    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca" pod "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" (UID: "7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:12.748585 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.748497    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sjm4\" (UniqueName: \"kubernetes.io/projected/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-kube-api-access-9sjm4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:12.748585 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:12.748524    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:13.444023 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:13.443994    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"
Nov 03 06:07:13.444444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:13.444037    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj" event={"ID":"7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08","Type":"ContainerDied","Data":"2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218"}
Nov 03 06:07:13.444444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:13.444095    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c5098ef76b9beaf7db95c85069bd1fc6efaebb589cf87615d709bdb4ae76218"
Nov 03 06:07:13.696853 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:07:13.696709    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:07:18.027816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.021775    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"]
Nov 03 06:07:18.044444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044406    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9e66745e-7b47-4051-b859-d04978db963a" containerName="revoke-keys"
Nov 03 06:07:18.044444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044435    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e66745e-7b47-4051-b859-d04978db963a" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044468    4528 cpu_manager.go:414] "containerMap: removing container" podUID="07e8ca0e-7e80-4205-8a69-3b679fce4c3d" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044475    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07e8ca0e-7e80-4205-8a69-3b679fce4c3d" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044491    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044499    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044522    4528 cpu_manager.go:414] "containerMap: removing container" podUID="743e4996-6f32-412a-9d75-b0bab9479c7e" containerName="revoke-keys"
Nov 03 06:07:18.044670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044529    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="743e4996-6f32-412a-9d75-b0bab9479c7e" containerName="revoke-keys"
Nov 03 06:07:18.045016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044677    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" containerName="revoke-keys"
Nov 03 06:07:18.045016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044697    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="743e4996-6f32-412a-9d75-b0bab9479c7e" containerName="revoke-keys"
Nov 03 06:07:18.045016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044707    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e66745e-7b47-4051-b859-d04978db963a" containerName="revoke-keys"
Nov 03 06:07:18.045016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.044722    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="07e8ca0e-7e80-4205-8a69-3b679fce4c3d" containerName="revoke-keys"
Nov 03 06:07:18.056119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.056084    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.093535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.081080    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"]
Nov 03 06:07:18.093535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.091435    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:07:18.252833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.252267    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:07:18.252833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.252413    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:07:18.260838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.254675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.260838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.254726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdx4\" (UniqueName: \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.260838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.254966    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.276885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.263829    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:07:18.346819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.335939    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:07:18.366055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.366020    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.369509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.369470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.381112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381074    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.381402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381379    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.381570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381553    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdx4\" (UniqueName: \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.381703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381688    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.381897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381869    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.382092 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.382073    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgw8c\" (UniqueName: \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.382277 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.381964    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-ba6892\"/\"distinct-w5-ba6892-kserve-self-signed-certs\""
Nov 03 06:07:18.382393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.382300    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.382523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.382340    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.382822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.382785    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.385131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.385075    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:07:18.458818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.440701    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdx4\" (UniqueName: \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.461930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.446192    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4920d59d-swvn6\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506539    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506711    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506742    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506770    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgw8c\" (UniqueName: \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.507387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.506849    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.509897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.509132    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.509897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.509389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.518032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.518006    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.521144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.521098    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.552843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.536410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.552843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.540429    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgw8c\" (UniqueName: \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\") pod \"distinct-w5-ba6892-kserve-fb9f786bd-ffrbn\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") " pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:18.585826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.583986    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:18.731820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:18.730886    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:19.042660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.042628    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"]
Nov 03 06:07:19.048505 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:19.048470    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode83b0d25_d8b3_419c_838d_dd93506f0910.slice/crio-2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691 WatchSource:0}: Error finding container 2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691: Status 404 returned error can't find the container with id 2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691
Nov 03 06:07:19.230181 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:19.230145    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod00f97b2e_cdd4_46f8_a4ec_a573c06e794c.slice/crio-7c1be09e5dc9acf40edf6d57efee4ae9c609b0fc42708173f94c9c4ecccb66c5 WatchSource:0}: Error finding container 7c1be09e5dc9acf40edf6d57efee4ae9c609b0fc42708173f94c9c4ecccb66c5: Status 404 returned error can't find the container with id 7c1be09e5dc9acf40edf6d57efee4ae9c609b0fc42708173f94c9c4ecccb66c5
Nov 03 06:07:19.230729 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.230631    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:07:19.528224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.528120    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" event={"ID":"00f97b2e-cdd4-46f8-a4ec-a573c06e794c","Type":"ContainerStarted","Data":"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"}
Nov 03 06:07:19.528224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.528171    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" event={"ID":"00f97b2e-cdd4-46f8-a4ec-a573c06e794c","Type":"ContainerStarted","Data":"7c1be09e5dc9acf40edf6d57efee4ae9c609b0fc42708173f94c9c4ecccb66c5"}
Nov 03 06:07:19.530040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.530010    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6" event={"ID":"e83b0d25-d8b3-419c-838d-dd93506f0910","Type":"ContainerStarted","Data":"4f60f0e7656e3c0d702183e4474a00c8fbb78d4208c9a5a19f99d6214bbe360a"}
Nov 03 06:07:19.530153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.530066    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6" event={"ID":"e83b0d25-d8b3-419c-838d-dd93506f0910","Type":"ContainerStarted","Data":"2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691"}
Nov 03 06:07:19.571768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.571716    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podStartSLOduration=1.571702497 podStartE2EDuration="1.571702497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:19.56870085 +0000 UTC m=+727.980097930" watchObservedRunningTime="2026-11-03 06:07:19.571702497 +0000 UTC m=+727.983099617"
Nov 03 06:07:19.613122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.613068    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6" podStartSLOduration=2.613054925 podStartE2EDuration="2.613054925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:19.608432191 +0000 UTC m=+728.019829282" watchObservedRunningTime="2026-11-03 06:07:19.613054925 +0000 UTC m=+728.024452007"
Nov 03 06:07:19.755554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:19.755525    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:20.541414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:20.541372    4528 generic.go:356] "Generic (PLEG): container finished" podID="e83b0d25-d8b3-419c-838d-dd93506f0910" containerID="4f60f0e7656e3c0d702183e4474a00c8fbb78d4208c9a5a19f99d6214bbe360a" exitCode=0
Nov 03 06:07:20.541920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:20.541457    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6" event={"ID":"e83b0d25-d8b3-419c-838d-dd93506f0910","Type":"ContainerDied","Data":"4f60f0e7656e3c0d702183e4474a00c8fbb78d4208c9a5a19f99d6214bbe360a"}
Nov 03 06:07:21.692465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.692437    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:21.791502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.791470    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\") pod \"e83b0d25-d8b3-419c-838d-dd93506f0910\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") "
Nov 03 06:07:21.791695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.791639    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\" (UniqueName: \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\") pod \"e83b0d25-d8b3-419c-838d-dd93506f0910\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") "
Nov 03 06:07:21.791767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.791695    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\") pod \"e83b0d25-d8b3-419c-838d-dd93506f0910\" (UID: \"e83b0d25-d8b3-419c-838d-dd93506f0910\") "
Nov 03 06:07:21.791894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.791866    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca" pod "e83b0d25-d8b3-419c-838d-dd93506f0910" (UID: "e83b0d25-d8b3-419c-838d-dd93506f0910"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:21.792200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.792175    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:21.793749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.793724    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4" pod "e83b0d25-d8b3-419c-838d-dd93506f0910" (UID: "e83b0d25-d8b3-419c-838d-dd93506f0910"). InnerVolumeSpecName "kube-api-access-vzdx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:21.793869 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.793815    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca" pod "e83b0d25-d8b3-419c-838d-dd93506f0910" (UID: "e83b0d25-d8b3-419c-838d-dd93506f0910"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:21.893538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.893463    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzdx4\" (UniqueName: \"kubernetes.io/projected/e83b0d25-d8b3-419c-838d-dd93506f0910-kube-api-access-vzdx4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:21.893538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:21.893498    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e83b0d25-d8b3-419c-838d-dd93506f0910-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:22.558966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:22.558935    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"
Nov 03 06:07:22.559143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:22.558934    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6" event={"ID":"e83b0d25-d8b3-419c-838d-dd93506f0910","Type":"ContainerDied","Data":"2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691"}
Nov 03 06:07:22.559143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:22.559064    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b5bf1556c7619fd4717397ff789121880dba0fac7e0291cface931944c26691"
Nov 03 06:07:23.402945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.402908    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"]
Nov 03 06:07:23.403812 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.403779    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e83b0d25-d8b3-419c-838d-dd93506f0910" containerName="revoke-keys"
Nov 03 06:07:23.403862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.403816    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e83b0d25-d8b3-419c-838d-dd93506f0910" containerName="revoke-keys"
Nov 03 06:07:23.404030 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.404001    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e83b0d25-d8b3-419c-838d-dd93506f0910" containerName="revoke-keys"
Nov 03 06:07:23.408435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.408407    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.425470 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.425433    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"]
Nov 03 06:07:23.429706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.429671    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:07:23.429706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.429677    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:07:23.429946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.429723    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:07:23.434005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.433973    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:07:23.514149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.514110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8rd9\" (UniqueName: \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.514369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.514341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.514412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.514392    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.616019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.615978    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8rd9\" (UniqueName: \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.616188 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.616116    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.616188 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.616138    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.616825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.616781    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.618755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.618724    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.639487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.639451    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8rd9\" (UniqueName: \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\") pod \"maas-api-revoke-sub-c2f9f128-fwwhk\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:23.726082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:23.725989    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:24.036545 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:24.036463    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c33d08_25fe_4fd2_a4d3_7dee88dcda33.slice/crio-c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7 WatchSource:0}: Error finding container c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7: Status 404 returned error can't find the container with id c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7
Nov 03 06:07:24.037414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:24.037386    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"]
Nov 03 06:07:24.586746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:24.586702    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk" event={"ID":"77c33d08-25fe-4fd2-a4d3-7dee88dcda33","Type":"ContainerStarted","Data":"d413667292ade9a06fd8028e40f0cca07abb12e08c270939c4e4de6d059b1630"}
Nov 03 06:07:24.587205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:24.586762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk" event={"ID":"77c33d08-25fe-4fd2-a4d3-7dee88dcda33","Type":"ContainerStarted","Data":"c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7"}
Nov 03 06:07:24.632949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:24.632889    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk" podStartSLOduration=1.6328742109999999 podStartE2EDuration="1.632874211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:24.630442329 +0000 UTC m=+733.041839411" watchObservedRunningTime="2026-11-03 06:07:24.632874211 +0000 UTC m=+733.044271290"
Nov 03 06:07:25.596894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:25.596847    4528 generic.go:356] "Generic (PLEG): container finished" podID="77c33d08-25fe-4fd2-a4d3-7dee88dcda33" containerID="d413667292ade9a06fd8028e40f0cca07abb12e08c270939c4e4de6d059b1630" exitCode=0
Nov 03 06:07:25.597327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:25.596925    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk" event={"ID":"77c33d08-25fe-4fd2-a4d3-7dee88dcda33","Type":"ContainerDied","Data":"d413667292ade9a06fd8028e40f0cca07abb12e08c270939c4e4de6d059b1630"}
Nov 03 06:07:26.562909 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.562867    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"]
Nov 03 06:07:26.567167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.567138    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.597486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.597453    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"]
Nov 03 06:07:26.638516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.638481    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"]
Nov 03 06:07:26.644166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.644131    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.651857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.651404    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"]
Nov 03 06:07:26.756848 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.756807    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.756957    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.756994    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757021    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757069    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757107    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757230    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757295    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757348    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757374    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138857e2-d465-4298-a8af-7ebfbab95b86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757518 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757496    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757545    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138857e2-d465-4298-a8af-7ebfbab95b86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757679    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757733    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx29n\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-kube-api-access-qx29n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757812    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.757899 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757880    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7vqk\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-kube-api-access-w7vqk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.757950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.757933    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.830024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.829959    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:26.859614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859578    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.859831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859630    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.859831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.859831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859679    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138857e2-d465-4298-a8af-7ebfbab95b86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.859831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859708    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.859831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859747    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138857e2-d465-4298-a8af-7ebfbab95b86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859890    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx29n\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-kube-api-access-qx29n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.859966    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7vqk\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-kube-api-access-w7vqk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860012    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860072    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860129    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860149    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860158    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860177    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860219    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860217    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-certs\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860229    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860244    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860270    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860298    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.860411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860311    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.861022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860425    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-workload-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.861022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-credential-socket\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.861022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-data\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.861022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.860958    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/138857e2-d465-4298-a8af-7ebfbab95b86-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.861223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.861135    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.862765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.862742    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/138857e2-d465-4298-a8af-7ebfbab95b86-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.862914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.862883    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.863027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.862993    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-envoy\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.863168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.863150    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/138857e2-d465-4298-a8af-7ebfbab95b86-istio-podinfo\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.886085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.886048    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx29n\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-kube-api-access-qx29n\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.888613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.888580    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/138857e2-d465-4298-a8af-7ebfbab95b86-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-q2zk7\" (UID: \"138857e2-d465-4298-a8af-7ebfbab95b86\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:26.889085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.889060    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-istio-token\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.889153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.889080    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7vqk\" (UniqueName: \"kubernetes.io/projected/bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb-kube-api-access-w7vqk\") pod \"maas-default-gateway-openshift-default-778b7b5c47-zw8r9\" (UID: \"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:26.961861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.961817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\") pod \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") "
Nov 03 06:07:26.962050 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.961915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\") pod \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") "
Nov 03 06:07:26.962106 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.962050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\" (UniqueName: \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\") pod \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\" (UID: \"77c33d08-25fe-4fd2-a4d3-7dee88dcda33\") "
Nov 03 06:07:26.962346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.962309    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca" pod "77c33d08-25fe-4fd2-a4d3-7dee88dcda33" (UID: "77c33d08-25fe-4fd2-a4d3-7dee88dcda33"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:26.962652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.962593    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:26.964086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.964065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9" pod "77c33d08-25fe-4fd2-a4d3-7dee88dcda33" (UID: "77c33d08-25fe-4fd2-a4d3-7dee88dcda33"). InnerVolumeSpecName "kube-api-access-w8rd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:26.964140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.964092    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca" pod "77c33d08-25fe-4fd2-a4d3-7dee88dcda33" (UID: "77c33d08-25fe-4fd2-a4d3-7dee88dcda33"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:26.966303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:26.966280    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:27.064133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.064092    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8rd9\" (UniqueName: \"kubernetes.io/projected/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-kube-api-access-w8rd9\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:27.064133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.064123    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77c33d08-25fe-4fd2-a4d3-7dee88dcda33-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:27.182755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.182652    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:27.293444 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:27.293392    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138857e2_d465_4298_a8af_7ebfbab95b86.slice/crio-053ab1f4d185a9b811aad00e6181a6021b856d38c659c9d444eb03865871856e WatchSource:0}: Error finding container 053ab1f4d185a9b811aad00e6181a6021b856d38c659c9d444eb03865871856e: Status 404 returned error can't find the container with id 053ab1f4d185a9b811aad00e6181a6021b856d38c659c9d444eb03865871856e
Nov 03 06:07:27.296813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.296756    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.296932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.296857    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.296932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.296904    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.299870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.299843    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"]
Nov 03 06:07:27.501680 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:27.501640    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdae4c2a_d4ca_4749_8c88_a62fbe20a3bb.slice/crio-0bd6a8651cf81cdd324ee5ae9896113260893d39f06e55ea2a5c34696536cb12 WatchSource:0}: Error finding container 0bd6a8651cf81cdd324ee5ae9896113260893d39f06e55ea2a5c34696536cb12: Status 404 returned error can't find the container with id 0bd6a8651cf81cdd324ee5ae9896113260893d39f06e55ea2a5c34696536cb12
Nov 03 06:07:27.503150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.503118    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"]
Nov 03 06:07:27.505096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.505056    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.505184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.505148    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.505229 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.505191    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:27.637811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.634869    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"
Nov 03 06:07:27.637811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.634886    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk" event={"ID":"77c33d08-25fe-4fd2-a4d3-7dee88dcda33","Type":"ContainerDied","Data":"c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7"}
Nov 03 06:07:27.637811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.634944    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4cd83fcb64b3c1e3ce3ccd193176df70cb48ece66e0d227f4b7172741f283d7"
Nov 03 06:07:27.642419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.642380    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" event={"ID":"138857e2-d465-4298-a8af-7ebfbab95b86","Type":"ContainerStarted","Data":"96cbb3bc2bb089489434df35d34ce05838799bb614db35e5bc7c6855f0487288"}
Nov 03 06:07:27.642580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.642453    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" event={"ID":"138857e2-d465-4298-a8af-7ebfbab95b86","Type":"ContainerStarted","Data":"053ab1f4d185a9b811aad00e6181a6021b856d38c659c9d444eb03865871856e"}
Nov 03 06:07:27.645500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.645471    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" event={"ID":"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb","Type":"ContainerStarted","Data":"810d1a113d11c225e3d85373dce592890b191d9a21e1db5cd2dcc7977582b2fd"}
Nov 03 06:07:27.645642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.645524    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" event={"ID":"bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb","Type":"ContainerStarted","Data":"0bd6a8651cf81cdd324ee5ae9896113260893d39f06e55ea2a5c34696536cb12"}
Nov 03 06:07:27.689260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.689196    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podStartSLOduration=1.689177353 podStartE2EDuration="1.689177353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:27.685365807 +0000 UTC m=+736.096762897" watchObservedRunningTime="2026-11-03 06:07:27.689177353 +0000 UTC m=+736.100574430"
Nov 03 06:07:27.748029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.747931    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podStartSLOduration=1.747916547 podStartE2EDuration="1.747916547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:27.743719871 +0000 UTC m=+736.155116950" watchObservedRunningTime="2026-11-03 06:07:27.747916547 +0000 UTC m=+736.159313624"
Nov 03 06:07:27.966840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:27.966810    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:28.044392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.044294    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"]
Nov 03 06:07:28.045320 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.045290    4528 cpu_manager.go:414] "containerMap: removing container" podUID="77c33d08-25fe-4fd2-a4d3-7dee88dcda33" containerName="revoke-keys"
Nov 03 06:07:28.045320 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.045321    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77c33d08-25fe-4fd2-a4d3-7dee88dcda33" containerName="revoke-keys"
Nov 03 06:07:28.045552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.045493    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="77c33d08-25fe-4fd2-a4d3-7dee88dcda33" containerName="revoke-keys"
Nov 03 06:07:28.050171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.050144    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.063388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.063354    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"]
Nov 03 06:07:28.065155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.065019    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:07:28.065155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.065151    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:07:28.065292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.065225    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:07:28.065341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.065299    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:07:28.177805 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.177753    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.177986 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.177968    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gwlk\" (UniqueName: \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.178029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.178019    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.183877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.183842    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:28.186341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.186306    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:28.186475 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.186363    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:28.280126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.280071    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.281745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.281708    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.281986 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.281964    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gwlk\" (UniqueName: \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.282143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.282128    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.306817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.305356    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.315077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.315003    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gwlk\" (UniqueName: \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\") pod \"maas-api-revoke-sub-de28d637-qdltd\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.380055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.380012    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:28.755737 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.755701    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:28.773379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.773348    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:28.834958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.834892    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"]
Nov 03 06:07:28.846703 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:28.846665    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82c35167_853e_43a2_b8c8_728174c0d3ab.slice/crio-fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b WatchSource:0}: Error finding container fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b: Status 404 returned error can't find the container with id fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b
Nov 03 06:07:28.969261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.969225    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:28.969460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:28.969297    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:29.215833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.200920    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:29.215833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.200990    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:29.835878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.835835    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd" event={"ID":"82c35167-853e-43a2-b8c8-728174c0d3ab","Type":"ContainerStarted","Data":"bd9232f5e4ec9400b5eebabdb4ff20fbbb0ee3c66d142472789f3e3868ad5be9"}
Nov 03 06:07:29.836469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.836445    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd" event={"ID":"82c35167-853e-43a2-b8c8-728174c0d3ab","Type":"ContainerStarted","Data":"fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b"}
Nov 03 06:07:29.910820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.909867    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd" podStartSLOduration=1.909844465 podStartE2EDuration="1.909844465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:29.887247802 +0000 UTC m=+738.298644883" watchObservedRunningTime="2026-11-03 06:07:29.909844465 +0000 UTC m=+738.321241547"
Nov 03 06:07:29.968814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.968759    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:29.969121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:29.969089    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:30.116473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.106448    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:07:30.236822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.198953    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:30.236822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.199012    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:30.877815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.870840    4528 generic.go:356] "Generic (PLEG): container finished" podID="82c35167-853e-43a2-b8c8-728174c0d3ab" containerID="bd9232f5e4ec9400b5eebabdb4ff20fbbb0ee3c66d142472789f3e3868ad5be9" exitCode=0
Nov 03 06:07:30.877815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.872189    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd" event={"ID":"82c35167-853e-43a2-b8c8-728174c0d3ab","Type":"ContainerDied","Data":"bd9232f5e4ec9400b5eebabdb4ff20fbbb0ee3c66d142472789f3e3868ad5be9"}
Nov 03 06:07:30.968738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.968265    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:30.968738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:30.968334    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:31.225873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:31.224948    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:31.225873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:31.225010    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:31.967080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:31.966939    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:31.967080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:31.967030    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:32.195044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.194945    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:32.195307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.195275    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:32.362653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.362619    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:32.471708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.471608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\") pod \"82c35167-853e-43a2-b8c8-728174c0d3ab\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") "
Nov 03 06:07:32.472012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.471727    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\") pod \"82c35167-853e-43a2-b8c8-728174c0d3ab\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") "
Nov 03 06:07:32.472012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.471821    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\" (UniqueName: \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\") pod \"82c35167-853e-43a2-b8c8-728174c0d3ab\" (UID: \"82c35167-853e-43a2-b8c8-728174c0d3ab\") "
Nov 03 06:07:32.473132 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.473101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca" pod "82c35167-853e-43a2-b8c8-728174c0d3ab" (UID: "82c35167-853e-43a2-b8c8-728174c0d3ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:32.474826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.474771    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca" pod "82c35167-853e-43a2-b8c8-728174c0d3ab" (UID: "82c35167-853e-43a2-b8c8-728174c0d3ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:32.479847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.475760    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk" pod "82c35167-853e-43a2-b8c8-728174c0d3ab" (UID: "82c35167-853e-43a2-b8c8-728174c0d3ab"). InnerVolumeSpecName "kube-api-access-8gwlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:32.573914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.573872    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:32.574072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.573934    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82c35167-853e-43a2-b8c8-728174c0d3ab-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:32.574072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.573944    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8gwlk\" (UniqueName: \"kubernetes.io/projected/82c35167-853e-43a2-b8c8-728174c0d3ab-kube-api-access-8gwlk\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:32.944858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.944236    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd" event={"ID":"82c35167-853e-43a2-b8c8-728174c0d3ab","Type":"ContainerDied","Data":"fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b"}
Nov 03 06:07:32.944858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.944268    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"
Nov 03 06:07:32.944858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.944303    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc6c27355fbb60abaf26d87c880a67b808a47806219ba0ffb57f12116a056d3b"
Nov 03 06:07:32.966853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.966821    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:32.966992 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:32.966873    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:33.184819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:33.183847    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:33.184819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:33.183925    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:33.967040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:33.966999    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:33.967238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:33.967060    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:34.183641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:34.183596    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:34.183888 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:34.183667    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:34.966993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:34.966952    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused" start-of-body=
Nov 03 06:07:34.967414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:34.967019    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": dial tcp 10.132.0.94:15021: connect: connection refused"
Nov 03 06:07:35.183865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:35.183823    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused" start-of-body=
Nov 03 06:07:35.184063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:35.183892    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": dial tcp 10.132.0.93:15021: connect: connection refused"
Nov 03 06:07:36.653366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.653329    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:36.653880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.653846    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:36.655149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.655128    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9"
Nov 03 06:07:36.802458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.802411    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:36.803211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.803187    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:36.803683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:36.803667    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7"
Nov 03 06:07:38.746925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.746892    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:07:38.747519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.747505    4528 cpu_manager.go:414] "containerMap: removing container" podUID="82c35167-853e-43a2-b8c8-728174c0d3ab" containerName="revoke-keys"
Nov 03 06:07:38.747558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.747519    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82c35167-853e-43a2-b8c8-728174c0d3ab" containerName="revoke-keys"
Nov 03 06:07:38.747616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.747607    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="82c35167-853e-43a2-b8c8-728174c0d3ab" containerName="revoke-keys"
Nov 03 06:07:38.752484 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.752460    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.785613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.785578    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:07:38.807719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.807675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.807951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.807870    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.807951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.807916    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.807948    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85wmt\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808146 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.808099    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.808151    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.808198    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.808225    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.808322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.808253    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909527 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909391    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909527 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909527 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909450    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85wmt\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909764    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909808    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909921    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909893    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.909998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.909973    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.910250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.910231    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.910430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.910411    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.910507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.910491    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.912148 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.912125    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.912416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.912397    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.922731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.922702    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:07:38.928312 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.928257    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:38.936724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.936695    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.937472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.937447    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85wmt\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:38.948473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:38.948441    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:07:39.011928 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.011822    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.011928 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.011888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.011928 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.011919    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.011953    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.012043    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.012091    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv2c5\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.012153    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.012238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.012418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.012304    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.074020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.073974    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:39.113658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113624    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.113874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113672    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv2c5\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.113874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113720    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.113874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113771    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.113874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113845    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113918    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113944    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113967    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.113994    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.114117    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.114358    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.114494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.114375    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.115022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.114986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.115140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.115117    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.117141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.117115    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.117991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.117968    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.139307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.139273    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.139454 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.139355    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv2c5\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.267773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.267652    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:39.405078 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:39.405029    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5589a2_bb5c_4eb0_9931_b383d8f37850.slice/crio-df3d13686c483f676bd6f29097094c2c1be7afa7a01c797d18016a3aeb89fd65 WatchSource:0}: Error finding container df3d13686c483f676bd6f29097094c2c1be7afa7a01c797d18016a3aeb89fd65: Status 404 returned error can't find the container with id df3d13686c483f676bd6f29097094c2c1be7afa7a01c797d18016a3aeb89fd65
Nov 03 06:07:39.407914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.407877    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:39.408005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.407949    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:39.408005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.407981    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:39.408603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.408574    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:07:39.590601 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:39.590568    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f0dbecd_a3c7_4168_a28a_8374e88b13f1.slice/crio-0d6c443fdf184d5ccf4867fc82d7d3ecdc6709f4bc43a5cb1de0970ea66d3fd1 WatchSource:0}: Error finding container 0d6c443fdf184d5ccf4867fc82d7d3ecdc6709f4bc43a5cb1de0970ea66d3fd1: Status 404 returned error can't find the container with id 0d6c443fdf184d5ccf4867fc82d7d3ecdc6709f4bc43a5cb1de0970ea66d3fd1
Nov 03 06:07:39.591231 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.591208    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:07:39.593468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.593434    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:39.593549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.593503    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:39.593549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:39.593539    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:40.025730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.025642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" event={"ID":"9c5589a2-bb5c-4eb0-9931-b383d8f37850","Type":"ContainerStarted","Data":"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"}
Nov 03 06:07:40.025730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.025708    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" event={"ID":"9c5589a2-bb5c-4eb0-9931-b383d8f37850","Type":"ContainerStarted","Data":"df3d13686c483f676bd6f29097094c2c1be7afa7a01c797d18016a3aeb89fd65"}
Nov 03 06:07:40.027629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.027600    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" event={"ID":"0f0dbecd-a3c7-4168-a28a-8374e88b13f1","Type":"ContainerStarted","Data":"54dfdb2772861492384834b8f69b685d3338e9488c54814626bf38252349933b"}
Nov 03 06:07:40.027745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.027653    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" event={"ID":"0f0dbecd-a3c7-4168-a28a-8374e88b13f1","Type":"ContainerStarted","Data":"0d6c443fdf184d5ccf4867fc82d7d3ecdc6709f4bc43a5cb1de0970ea66d3fd1"}
Nov 03 06:07:40.075606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.075569    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:40.075813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.075740    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podStartSLOduration=2.075724721 podStartE2EDuration="2.075724721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:40.073405016 +0000 UTC m=+748.484802107" watchObservedRunningTime="2026-11-03 06:07:40.075724721 +0000 UTC m=+748.487121801"
Nov 03 06:07:40.090819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.079908    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:40.090819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.079965    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:40.129161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.129081    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podStartSLOduration=2.129059736 podStartE2EDuration="2.129059736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:40.121438973 +0000 UTC m=+748.532836050" watchObservedRunningTime="2026-11-03 06:07:40.129059736 +0000 UTC m=+748.540456815"
Nov 03 06:07:40.268466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.268431    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:40.270288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.270255    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:40.270434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:40.270322    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:41.074987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:41.074948    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:41.075399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:41.075006    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:41.269000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:41.268965    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:41.269174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:41.269026    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:42.075249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:42.075216    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:42.075706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:42.075275    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:42.268119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:42.268087    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:42.268285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:42.268136    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:43.075360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.075320    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:43.075739 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.075391    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:43.268666 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.268629    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:43.268879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.268684    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:43.528938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.528849    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:07:43.543626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.543599    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.552074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.552045    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:07:43.567497 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.567461    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"default-dockercfg-nhbrc\""
Nov 03 06:07:43.569137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.567761    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"distinct-2-w2-5c3542-kserve-self-signed-certs\""
Nov 03 06:07:43.570661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.570641    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"kube-root-ca.crt\""
Nov 03 06:07:43.570782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.570719    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-5c3542\"/\"openshift-service-ca.crt\""
Nov 03 06:07:43.639699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.639659    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.639904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.639882    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.639964 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.639940    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.640146 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.640116    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.640235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.640164    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.640235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.640201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4m6m\" (UniqueName: \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.768608    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.768658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.768681    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4m6m\" (UniqueName: \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.768785    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.768968    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.769817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.769017    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.777108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.776632    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.798346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.797892    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.798346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.798136    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.869498 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:07:43.869450    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:07:43.874341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.874295    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"]
Nov 03 06:07:43.946067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.944216    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.946067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.944958    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:43.947918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.947863    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.948595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.948574    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4m6m\" (UniqueName: \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\") pod \"distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") " pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:43.955444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.955233    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"]
Nov 03 06:07:43.959911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.959880    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:07:43.960005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.959914    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:07:43.960005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.959947    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:07:43.960104 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:43.960088    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:07:44.048039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.048002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.048230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.048200    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.048315 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.048299    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmmtv\" (UniqueName: \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.074355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.074319    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:44.074505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.074375    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:44.149744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.149709    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.150136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.149827    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmmtv\" (UniqueName: \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.150136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.149850    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.150727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.150702    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.152299 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.152276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.161481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.161458    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:44.174437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.174386    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmmtv\" (UniqueName: \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\") pod \"maas-api-revoke-sub-bbecaa07-prdpc\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.266221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.263652    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:44.270563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.270417    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:44.270563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.270485    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:44.650854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.650332    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:07:44.748546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:44.748506    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"]
Nov 03 06:07:44.756371 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:44.756333    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod257c6eaa_f097_4883_8619_523303809c4c.slice/crio-1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f WatchSource:0}: Error finding container 1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f: Status 404 returned error can't find the container with id 1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f
Nov 03 06:07:45.075699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.075653    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:45.083951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.075725    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:45.153818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.153131    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc" event={"ID":"257c6eaa-f097-4883-8619-523303809c4c","Type":"ContainerStarted","Data":"1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f"}
Nov 03 06:07:45.155126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.155099    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" event={"ID":"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67","Type":"ContainerStarted","Data":"fce25e2c7f6f170f3f216a9471c7a3b99f77d9b5856725b47852efb5087c270c"}
Nov 03 06:07:45.155278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.155266    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" event={"ID":"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67","Type":"ContainerStarted","Data":"489c7679dedb3aedc21d97423ef70a5577f47545a57c994d800352a81d758eb0"}
Nov 03 06:07:45.162555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.162528    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:45.239819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.227333    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podStartSLOduration=2.227308543 podStartE2EDuration="2.227308543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:45.200274271 +0000 UTC m=+753.611671350" watchObservedRunningTime="2026-11-03 06:07:45.227308543 +0000 UTC m=+753.638705622"
Nov 03 06:07:45.268434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.268347    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body=
Nov 03 06:07:45.268434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:45.268392    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused"
Nov 03 06:07:46.075071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.075028    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body=
Nov 03 06:07:46.075280 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.075083    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused"
Nov 03 06:07:46.169647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.169552    4528 generic.go:356] "Generic (PLEG): container finished" podID="257c6eaa-f097-4883-8619-523303809c4c" containerID="1f8a94f7ee22fa2a917dc0f91ae189f615bb7001cd46aa8fb54bdadb486a7a1f" exitCode=0
Nov 03 06:07:46.170318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.170290    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc" event={"ID":"257c6eaa-f097-4883-8619-523303809c4c","Type":"ContainerDied","Data":"1f8a94f7ee22fa2a917dc0f91ae189f615bb7001cd46aa8fb54bdadb486a7a1f"}
Nov 03 06:07:46.566288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.566017    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:46.570849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.567185    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:46.570849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:46.569392    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:07:47.080580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.080542    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:47.081069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.081045    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:47.081934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.081910    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:07:47.377302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.377273    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:47.448678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.448631    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\" (UniqueName: \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\") pod \"257c6eaa-f097-4883-8619-523303809c4c\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") "
Nov 03 06:07:47.448897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.448875    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\") pod \"257c6eaa-f097-4883-8619-523303809c4c\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") "
Nov 03 06:07:47.449024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.448957    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\") pod \"257c6eaa-f097-4883-8619-523303809c4c\" (UID: \"257c6eaa-f097-4883-8619-523303809c4c\") "
Nov 03 06:07:47.449243 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.449213    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca" pod "257c6eaa-f097-4883-8619-523303809c4c" (UID: "257c6eaa-f097-4883-8619-523303809c4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:07:47.449588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.449569    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/257c6eaa-f097-4883-8619-523303809c4c-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:47.451088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.451067    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca" pod "257c6eaa-f097-4883-8619-523303809c4c" (UID: "257c6eaa-f097-4883-8619-523303809c4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:07:47.451463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.451427    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv" pod "257c6eaa-f097-4883-8619-523303809c4c" (UID: "257c6eaa-f097-4883-8619-523303809c4c"). InnerVolumeSpecName "kube-api-access-gmmtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:07:47.551258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.551219    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/257c6eaa-f097-4883-8619-523303809c4c-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:47.551258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:47.551257    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmmtv\" (UniqueName: \"kubernetes.io/projected/257c6eaa-f097-4883-8619-523303809c4c-kube-api-access-gmmtv\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:07:48.190000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:48.189970    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"
Nov 03 06:07:48.190231 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:48.189998    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc" event={"ID":"257c6eaa-f097-4883-8619-523303809c4c","Type":"ContainerDied","Data":"1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f"}
Nov 03 06:07:48.190231 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:48.190052    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fc3dee5f17d5b8b6130b52b9791c8d0a02251bd66515d8ee0ffaefc5ef5ce1f"
Nov 03 06:07:52.363562 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.363525    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:07:52.364379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.364355    4528 cpu_manager.go:414] "containerMap: removing container" podUID="257c6eaa-f097-4883-8619-523303809c4c" containerName="revoke-keys"
Nov 03 06:07:52.364379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.364378    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="257c6eaa-f097-4883-8619-523303809c4c" containerName="revoke-keys"
Nov 03 06:07:52.364573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.364557    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="257c6eaa-f097-4883-8619-523303809c4c" containerName="revoke-keys"
Nov 03 06:07:52.368641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.368616    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.395196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.391281    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:07:52.395196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.392676    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"default-dockercfg-xtbd8\""
Nov 03 06:07:52.395196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.392783    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"distinct-w4-be1a66-kserve-self-signed-certs\""
Nov 03 06:07:52.398191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.398164    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"openshift-service-ca.crt\""
Nov 03 06:07:52.398335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.398277    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-be1a66\"/\"kube-root-ca.crt\""
Nov 03 06:07:52.422204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422168    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.422414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422225    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.422414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422275    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.422414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422370    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.422584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422450    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.422584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.422481    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwz9x\" (UniqueName: \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.523980    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524030    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524166    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524311 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524297    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524390    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524475 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524424    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwz9x\" (UniqueName: \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524615    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524666    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.524778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.524757    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.526990 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.526963    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.527218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.527199    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.550521 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.550482    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwz9x\" (UniqueName: \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\") pod \"distinct-w4-be1a66-kserve-798875cfd-bcjzn\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") " pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:52.697705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:52.697613    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:53.021047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:53.021007    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:07:53.274067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:53.273977    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" event={"ID":"37e31898-dee0-4975-8b4d-8b8f851b223e","Type":"ContainerStarted","Data":"acb6887b86064bb0017e90f4c81d70e3c0ffa2b88069a4f205bf3b7e7e758936"}
Nov 03 06:07:53.274067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:53.274024    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" event={"ID":"37e31898-dee0-4975-8b4d-8b8f851b223e","Type":"ContainerStarted","Data":"2be26789a5d5d91f8cb5e5b1f889a646135460b0808f3328ff76ce2b9add7a34"}
Nov 03 06:07:53.316267 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:53.316206    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podStartSLOduration=1.3161907369999999 podStartE2EDuration="1.316190737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:53.314448634 +0000 UTC m=+761.725845714" watchObservedRunningTime="2026-11-03 06:07:53.316190737 +0000 UTC m=+761.727587808"
Nov 03 06:07:53.697935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:53.697834    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:07:54.052551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.052520    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:07:54.057412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.057387    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.071713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.071684    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-be1a66-openshift-default-dockercfg-kn8kd\""
Nov 03 06:07:54.089481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.089445    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:07:54.145553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145513    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145629    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145693    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145760    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8v4\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145826    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.145905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145894    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.146079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145967    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.146079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.145987    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.162364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.162329    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:54.175385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.175355    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:54.246894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.246855    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.246931    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247163    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247274    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247326    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8v4\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247335    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247407    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247477    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247566    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247600    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247761 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247740    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.247942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.247920    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.248124 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.248099    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.248276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.248255    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.249386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.249366    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.251167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.251150    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.271005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.270970    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8v4\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.271575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.271552    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.298364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.298328    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:07:54.374645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.374560    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:54.738744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.738715    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:07:54.739435 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:07:54.739410    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee16f136_f2d3_41e7_9075_dff10ab7f195.slice/crio-8a888a0f1fcf817314c60db63d6008db900f17787367f140aa31be93890da258 WatchSource:0}: Error finding container 8a888a0f1fcf817314c60db63d6008db900f17787367f140aa31be93890da258: Status 404 returned error can't find the container with id 8a888a0f1fcf817314c60db63d6008db900f17787367f140aa31be93890da258
Nov 03 06:07:54.742951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.742908    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:54.743049 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.742999    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:54.743121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:54.743044    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:07:55.337769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.337728    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" event={"ID":"ee16f136-f2d3-41e7-9075-dff10ab7f195","Type":"ContainerStarted","Data":"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"}
Nov 03 06:07:55.337955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.337831    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" event={"ID":"ee16f136-f2d3-41e7-9075-dff10ab7f195","Type":"ContainerStarted","Data":"8a888a0f1fcf817314c60db63d6008db900f17787367f140aa31be93890da258"}
Nov 03 06:07:55.375605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.375574    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:07:55.406818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.406586    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:07:55.406818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.406646    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:07:55.407083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:55.406935    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podStartSLOduration=1.40691794 podStartE2EDuration="1.40691794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:07:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:07:55.406376405 +0000 UTC m=+763.817773484" watchObservedRunningTime="2026-11-03 06:07:55.40691794 +0000 UTC m=+763.818315021"
Nov 03 06:07:56.375602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:56.375569    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:07:56.376142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:56.375619    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:07:57.375366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:57.375327    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:07:57.375552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:57.375384    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:07:58.375013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:58.374970    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:07:58.375509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:58.375038    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:07:59.375617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:59.375585    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:07:59.376110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:07:59.375636    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:08:00.411014 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:00.408311    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Nov 03 06:08:00.411014 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:00.408380    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Nov 03 06:08:01.513109 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:01.513072    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:08:01.514054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:01.514026    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:08:01.516753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:01.516724    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:08:02.698802 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:02.698745    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:08:02.711845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:02.711813    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:08:03.455354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:03.455317    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:08:07.811040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.810999    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"]
Nov 03 06:08:07.817629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.817601    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:07.841039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.841010    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:08:07.848109 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.841010    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:08:07.848344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.841104    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:08:07.849752 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.849676    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"]
Nov 03 06:08:07.852120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:07.852080    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:08:08.158683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.158578    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtb7l\" (UniqueName: \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.159177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.159150    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.159807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.159765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.262846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.262756    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.263216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.263190    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.263419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.263396    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtb7l\" (UniqueName: \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.264368 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.264248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.269020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.268998    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.302548 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.302515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtb7l\" (UniqueName: \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\") pod \"maas-api-revoke-sub-36f95078-czqdc\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:08.452200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:08.452093    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:09.194374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.192923    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"]
Nov 03 06:08:09.199565 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:09.199535    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3d6cd9_858d_407f_ac70_f39bf95f488e.slice/crio-bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896 WatchSource:0}: Error finding container bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896: Status 404 returned error can't find the container with id bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896
Nov 03 06:08:09.548615 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.548567    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc" event={"ID":"8c3d6cd9-858d-407f-ac70-f39bf95f488e","Type":"ContainerStarted","Data":"57849d59993c7623094a6a149a3ff9bdd1ff17d42471de55ac8d21b056a7f94f"}
Nov 03 06:08:09.548822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.548630    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc" event={"ID":"8c3d6cd9-858d-407f-ac70-f39bf95f488e","Type":"ContainerStarted","Data":"bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896"}
Nov 03 06:08:09.594433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.594367    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc" podStartSLOduration=2.594348007 podStartE2EDuration="2.594348007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:09.587468008 +0000 UTC m=+777.998865086" watchObservedRunningTime="2026-11-03 06:08:09.594348007 +0000 UTC m=+778.005745087"
Nov 03 06:08:09.710297 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.710257    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:08:09.716004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.715973    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.760614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.760532    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:08:09.770649 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770604    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.770843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770676    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.770843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770749    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.770843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770779    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trtcn\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.771007 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770855    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.771007 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770895    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.771007 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.770968    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.771156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.771018    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.771156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.771128    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872377    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872491    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872519    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trtcn\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872568    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872649    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872675    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.872890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.872753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.873270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.873241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.873377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.873290    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.873377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.873321    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.873637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.873612    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.873637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.873622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.875275 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.875245    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.875499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.875476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.895109 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.895078    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trtcn\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:09.895266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:09.895077    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:10.036878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.036756    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:10.368442 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:10.368400    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda019eebc_2834_4c31_b4df_b484cf5c4708.slice/crio-0181fa94ed5b2232dbe00d86abe3c15be9973d8373f1a29a3628742ac404d5a0 WatchSource:0}: Error finding container 0181fa94ed5b2232dbe00d86abe3c15be9973d8373f1a29a3628742ac404d5a0: Status 404 returned error can't find the container with id 0181fa94ed5b2232dbe00d86abe3c15be9973d8373f1a29a3628742ac404d5a0
Nov 03 06:08:10.370115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.370088    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:08:10.371640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.371597    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:10.371750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.371670    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:10.371750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.371704    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:10.561369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.561336    4528 generic.go:356] "Generic (PLEG): container finished" podID="8c3d6cd9-858d-407f-ac70-f39bf95f488e" containerID="57849d59993c7623094a6a149a3ff9bdd1ff17d42471de55ac8d21b056a7f94f" exitCode=0
Nov 03 06:08:10.561571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.561419    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc" event={"ID":"8c3d6cd9-858d-407f-ac70-f39bf95f488e","Type":"ContainerDied","Data":"57849d59993c7623094a6a149a3ff9bdd1ff17d42471de55ac8d21b056a7f94f"}
Nov 03 06:08:10.563300 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.563271    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" event={"ID":"a019eebc-2834-4c31-b4df-b484cf5c4708","Type":"ContainerStarted","Data":"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"}
Nov 03 06:08:10.563438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.563318    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" event={"ID":"a019eebc-2834-4c31-b4df-b484cf5c4708","Type":"ContainerStarted","Data":"0181fa94ed5b2232dbe00d86abe3c15be9973d8373f1a29a3628742ac404d5a0"}
Nov 03 06:08:10.647200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:10.647082    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podStartSLOduration=1.6470623070000001 podStartE2EDuration="1.647062307s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:10.64621331 +0000 UTC m=+779.057610390" watchObservedRunningTime="2026-11-03 06:08:10.647062307 +0000 UTC m=+779.058459384"
Nov 03 06:08:11.037093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:11.037006    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:11.038492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:11.038464    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:11.038617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:11.038515    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:11.950781 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:11.946295    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:12.037509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.037467    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:12.037708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.037528    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:12.126185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.126152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\") pod \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") "
Nov 03 06:08:12.126379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.126308    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\" (UniqueName: \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\") pod \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") "
Nov 03 06:08:12.126441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.126420    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\") pod \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\" (UID: \"8c3d6cd9-858d-407f-ac70-f39bf95f488e\") "
Nov 03 06:08:12.126815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.126762    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca" pod "8c3d6cd9-858d-407f-ac70-f39bf95f488e" (UID: "8c3d6cd9-858d-407f-ac70-f39bf95f488e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:08:12.127028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.127011    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:12.128669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.128642    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca" pod "8c3d6cd9-858d-407f-ac70-f39bf95f488e" (UID: "8c3d6cd9-858d-407f-ac70-f39bf95f488e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:08:12.129089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.129066    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l" pod "8c3d6cd9-858d-407f-ac70-f39bf95f488e" (UID: "8c3d6cd9-858d-407f-ac70-f39bf95f488e"). InnerVolumeSpecName "kube-api-access-gtb7l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:08:12.230614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.230568    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c3d6cd9-858d-407f-ac70-f39bf95f488e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:12.230927 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.230621    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtb7l\" (UniqueName: \"kubernetes.io/projected/8c3d6cd9-858d-407f-ac70-f39bf95f488e-kube-api-access-gtb7l\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:12.623150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.623117    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"
Nov 03 06:08:12.623375 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.623119    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc" event={"ID":"8c3d6cd9-858d-407f-ac70-f39bf95f488e","Type":"ContainerDied","Data":"bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896"}
Nov 03 06:08:12.623375 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:12.623246    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfffb253a35ee8d0c55a07db66327ccb47bb7d93a0f41a1558c6df088c292896"
Nov 03 06:08:13.037861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:13.037765    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:13.038346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:13.037852    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:13.714567 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:08:13.714522    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:08:14.038119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:14.038029    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:14.038119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:14.038087    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:15.038184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:15.038144    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:15.038600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:15.038200    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:16.038195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:16.038150    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused" start-of-body=
Nov 03 06:08:16.038690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:16.038220    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": dial tcp 10.132.0.103:15021: connect: connection refused"
Nov 03 06:08:17.359807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:17.359770    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:17.360655 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:17.360625    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:17.361679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:17.361655    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:08:21.839319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.839279    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"]
Nov 03 06:08:21.855058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.840858    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8c3d6cd9-858d-407f-ac70-f39bf95f488e" containerName="revoke-keys"
Nov 03 06:08:21.855058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.852746    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c3d6cd9-858d-407f-ac70-f39bf95f488e" containerName="revoke-keys"
Nov 03 06:08:21.855058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.853036    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c3d6cd9-858d-407f-ac70-f39bf95f488e" containerName="revoke-keys"
Nov 03 06:08:21.858888 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.858864    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"]
Nov 03 06:08:21.859041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.858976    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:21.873365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.873338    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:08:21.873479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.873403    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:08:21.927181 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.927130    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:08:21.927606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.927585    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:08:21.946532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.944510    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:21.946532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.944702    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld498\" (UniqueName: \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:21.946532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:21.944772    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.046091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.046040    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld498\" (UniqueName: \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.046292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.046123    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.046292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.046246    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.047249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.047220    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.048875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.048850    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.069785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.069752    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld498\" (UniqueName: \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\") pod \"maas-api-revoke-sub-9ef85531-s22tk\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.175517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.175433    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:22.528540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.528511    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"]
Nov 03 06:08:22.775435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:22.775347    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk" event={"ID":"a3ba9efb-1a22-45f8-b747-207be6f10fbe","Type":"ContainerStarted","Data":"9e36846205b3593c9b7bddd3f4d78fb1b89704f5ab7bde1b50dfd7a24f8d6f47"}
Nov 03 06:08:23.789191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:23.789098    4528 generic.go:356] "Generic (PLEG): container finished" podID="a3ba9efb-1a22-45f8-b747-207be6f10fbe" containerID="62980d1a48298286e294f2d71078b2b0822b5a35cc7324f40c701b8e951e57fe" exitCode=0
Nov 03 06:08:23.789618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:23.789181    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk" event={"ID":"a3ba9efb-1a22-45f8-b747-207be6f10fbe","Type":"ContainerDied","Data":"62980d1a48298286e294f2d71078b2b0822b5a35cc7324f40c701b8e951e57fe"}
Nov 03 06:08:24.044935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.044840    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:08:24.049608 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.049580    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.071202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.071168    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:08:24.076441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076406    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvd5f\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076532    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076566    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076607    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076680    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076780 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076731    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.076946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.076922    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.077038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.077022    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.077097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.077080    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178351    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178437    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178484    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178549    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178762 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178762 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178692    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178758    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvd5f\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178849    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.178913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178885    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.179266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.179241    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.179486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.179460    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.179567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.179476    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.179567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.178846    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.179880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.179854    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.183077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.183051    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.183557 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.183532    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.201946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.201919    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.202205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.202184    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvd5f\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\") pod \"e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") " pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.364591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.364556    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:24.416771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.416735    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:08:24.424245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.424218    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.451818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.448884    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:08:24.483152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483048    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483098    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483222    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483310    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483347    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483373    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483607 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483465    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483607 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483490    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.483607 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.483543    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.584979    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585023    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585069    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585225    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585304    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585399    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585632 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.585860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.585834    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.586284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.586159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.586284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.586208    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.586284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.586211    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.586284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.586232    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.588121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.588092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.588432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.588410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.614116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.614085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.614258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.614237    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.697719 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:24.697687    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75905cf_8c83_4133_a545_bcd5e4027069.slice/crio-259a6155d9c9c4ca443dc96ff51d424e9ccae79590bc37b209380be69dbaee7a WatchSource:0}: Error finding container 259a6155d9c9c4ca443dc96ff51d424e9ccae79590bc37b209380be69dbaee7a: Status 404 returned error can't find the container with id 259a6155d9c9c4ca443dc96ff51d424e9ccae79590bc37b209380be69dbaee7a
Nov 03 06:08:24.700398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.700359    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:24.700522 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.700449    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:24.700522 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.700497    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:24.700695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.700668    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:08:24.763076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.763043    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:24.807252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:24.807009    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" event={"ID":"a75905cf-8c83-4133-a545-bcd5e4027069","Type":"ContainerStarted","Data":"259a6155d9c9c4ca443dc96ff51d424e9ccae79590bc37b209380be69dbaee7a"}
Nov 03 06:08:25.035926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.035633    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:25.107080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.106885    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\" (UniqueName: \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\") pod \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") "
Nov 03 06:08:25.107621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.107559    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\") pod \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") "
Nov 03 06:08:25.107820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.107802    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\") pod \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\" (UID: \"a3ba9efb-1a22-45f8-b747-207be6f10fbe\") "
Nov 03 06:08:25.108914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.108888    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca" pod "a3ba9efb-1a22-45f8-b747-207be6f10fbe" (UID: "a3ba9efb-1a22-45f8-b747-207be6f10fbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:08:25.111617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.111563    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca" pod "a3ba9efb-1a22-45f8-b747-207be6f10fbe" (UID: "a3ba9efb-1a22-45f8-b747-207be6f10fbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:08:25.116226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.116164    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498" pod "a3ba9efb-1a22-45f8-b747-207be6f10fbe" (UID: "a3ba9efb-1a22-45f8-b747-207be6f10fbe"). InnerVolumeSpecName "kube-api-access-ld498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:08:25.128917 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:25.128872    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c03e07e_1e7a_4ab6_a890_0a379fc2ad57.slice/crio-9e1bb24ddb277ebaf9f4d41d08d52a6aaefb46f5389a0409dc29e08e59a85ce1 WatchSource:0}: Error finding container 9e1bb24ddb277ebaf9f4d41d08d52a6aaefb46f5389a0409dc29e08e59a85ce1: Status 404 returned error can't find the container with id 9e1bb24ddb277ebaf9f4d41d08d52a6aaefb46f5389a0409dc29e08e59a85ce1
Nov 03 06:08:25.132368 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.132329    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:25.132462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.132418    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:25.132523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.132465    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:25.134281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.134250    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:08:25.209924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.209889    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:25.209924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.209922    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3ba9efb-1a22-45f8-b747-207be6f10fbe-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:25.210142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.209938    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld498\" (UniqueName: \"kubernetes.io/projected/a3ba9efb-1a22-45f8-b747-207be6f10fbe-kube-api-access-ld498\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:25.817784 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.817738    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" event={"ID":"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57","Type":"ContainerStarted","Data":"7e2ac5c2bcca9e6dd17f650486c3c79e820def2d7d3dedcd1d2a413004259cc8"}
Nov 03 06:08:25.818276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.817824    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" event={"ID":"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57","Type":"ContainerStarted","Data":"9e1bb24ddb277ebaf9f4d41d08d52a6aaefb46f5389a0409dc29e08e59a85ce1"}
Nov 03 06:08:25.819861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.819829    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" event={"ID":"a75905cf-8c83-4133-a545-bcd5e4027069","Type":"ContainerStarted","Data":"8d90ab093b2dc2211301cd1e4ca7618ab1cad0ea284743e1ce7c27e7af33ff6a"}
Nov 03 06:08:25.821494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.821471    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk" event={"ID":"a3ba9efb-1a22-45f8-b747-207be6f10fbe","Type":"ContainerDied","Data":"9e36846205b3593c9b7bddd3f4d78fb1b89704f5ab7bde1b50dfd7a24f8d6f47"}
Nov 03 06:08:25.821641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.821525    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e36846205b3593c9b7bddd3f4d78fb1b89704f5ab7bde1b50dfd7a24f8d6f47"
Nov 03 06:08:25.821641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.821568    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"
Nov 03 06:08:25.917816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.916918    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podStartSLOduration=1.916897681 podStartE2EDuration="1.916897681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:25.885281175 +0000 UTC m=+794.296678253" watchObservedRunningTime="2026-11-03 06:08:25.916897681 +0000 UTC m=+794.328294761"
Nov 03 06:08:25.952822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:25.952703    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podStartSLOduration=1.952684533 podStartE2EDuration="1.952684533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:25.948852469 +0000 UTC m=+794.360249548" watchObservedRunningTime="2026-11-03 06:08:25.952684533 +0000 UTC m=+794.364081611"
Nov 03 06:08:26.365683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.365651    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:26.366630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.366600    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:26.366684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.366656    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:26.763377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.763277    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:26.767102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.767075    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:26.767226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:26.767128    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:27.365208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:27.365173    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:27.365672 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:27.365233    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:27.764450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:27.764365    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:27.764450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:27.764416    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:28.370817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:28.370225    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:28.370817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:28.370286    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:28.763436 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:28.763352    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:28.763637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:28.763613    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:29.365230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:29.365199    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:29.365541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:29.365520    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:29.763501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:29.763422    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:29.763501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:29.763479    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:30.365019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:30.364988    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:30.365261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:30.365238    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:30.764031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:30.763940    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:30.764031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:30.763996    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:31.365532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:31.365494    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused" start-of-body=
Nov 03 06:08:31.365715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:31.365565    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": dial tcp 10.132.0.105:15021: connect: connection refused"
Nov 03 06:08:31.763932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:31.763778    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused" start-of-body=
Nov 03 06:08:31.763932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:31.763914    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": dial tcp 10.132.0.106:15021: connect: connection refused"
Nov 03 06:08:32.370281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.370246    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:32.370773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.370741    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:32.371506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.371482    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:08:32.768491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.768407    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:32.768977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.768950    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:32.769628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:32.769610    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:08:38.788718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.788684    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:08:38.790406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.790382    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a3ba9efb-1a22-45f8-b747-207be6f10fbe" containerName="revoke-keys"
Nov 03 06:08:38.790406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.790405    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3ba9efb-1a22-45f8-b747-207be6f10fbe" containerName="revoke-keys"
Nov 03 06:08:38.790606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.790547    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3ba9efb-1a22-45f8-b747-207be6f10fbe" containerName="revoke-keys"
Nov 03 06:08:38.795069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.795032    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.823870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.820396    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:08:38.880746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.880707    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.880965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.880769    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.880965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.880823    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.880997    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.881068    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl2kq\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.881244    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.881391    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.881420    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.881586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.881563    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982518    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl2kq\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982650    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982769    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.982889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982878    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982899    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982935    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.982978    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.983011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.983174    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.983423    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.983640    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.983973 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.983949    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.985643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.985617    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:38.985839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:38.985818    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:39.006301 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.006271    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl2kq\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:39.006597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.006572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:39.121565 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.121529    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:39.451591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.451555    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:08:39.454384 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.454346    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:39.454492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.454411    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:39.454492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.454440    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:39.515215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.515180    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:08:39.522458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.522433    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.544223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.544191    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:08:39.558546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.557882    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:08:39.564366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.564337    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.603814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.603465    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.699767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.699854    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.699918    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.699944    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psbn7\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.699976    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700040    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700078    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700108    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700147    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700183    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700226    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700256    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700293    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700319    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700390    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdcc\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700551    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.700877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.700592    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.801556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801514    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801571    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801604    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801641    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801700    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801722    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psbn7\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801753    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801856    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801898    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801929    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.801970    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802001    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802044    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802077    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802092    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802113    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802142    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802167    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802182    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802215    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdcc\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.802555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802497    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.803057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.802986    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.803507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.803148    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.803507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.803268    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.803507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.803456    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.804021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.803773    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.804021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.803983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.804461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.804433    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.805310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.805265    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.805430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.805404    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.805524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.805448    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.806067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.806042    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.827117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.827071    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.827539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.827515    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdcc\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.827600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.827512    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psbn7\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.827942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.827918    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.838269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.838247    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:39.903361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.903327    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:39.977048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.976940    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" event={"ID":"8acc50f4-c6c7-412b-bdea-70f82ba0186f","Type":"ContainerStarted","Data":"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"}
Nov 03 06:08:39.977048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:39.977012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" event={"ID":"8acc50f4-c6c7-412b-bdea-70f82ba0186f","Type":"ContainerStarted","Data":"4629bcf4e2f05caa5a238f5e62392622edd14d09d5899274fbca41b735e7aba6"}
Nov 03 06:08:40.035334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.035106    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podStartSLOduration=2.035085854 podStartE2EDuration="2.035085854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:40.033822952 +0000 UTC m=+808.445220051" watchObservedRunningTime="2026-11-03 06:08:40.035085854 +0000 UTC m=+808.446482933"
Nov 03 06:08:40.122594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.122565    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:40.124004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.123903    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:40.124004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.123960    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:40.167936 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.167905    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:08:40.168842 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:40.168809    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80326b77_b409_4f1a_9deb_145840baaa82.slice/crio-a07396e578de25b4ac4a404d387598501e499c361c02ff33af9be8273ebf5ea0 WatchSource:0}: Error finding container a07396e578de25b4ac4a404d387598501e499c361c02ff33af9be8273ebf5ea0: Status 404 returned error can't find the container with id a07396e578de25b4ac4a404d387598501e499c361c02ff33af9be8273ebf5ea0
Nov 03 06:08:40.171980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.171945    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:40.172060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.172013    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:40.172060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.172044    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:40.263438 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:40.263407    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b2cff8e_1e9c_45d4_8ac1_34dfcb1cbcc3.slice/crio-32feec40a3f812ad1c84b6a3abf5d3d5a1f9cfc57049547dec43978445e092d7 WatchSource:0}: Error finding container 32feec40a3f812ad1c84b6a3abf5d3d5a1f9cfc57049547dec43978445e092d7: Status 404 returned error can't find the container with id 32feec40a3f812ad1c84b6a3abf5d3d5a1f9cfc57049547dec43978445e092d7
Nov 03 06:08:40.264344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.264313    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:08:40.266210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.266170    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:40.266299 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.266259    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:40.266348 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:40.266331    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:41.049817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.048321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" event={"ID":"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3","Type":"ContainerStarted","Data":"e86ee71739b61f7f13fdc4985dc96a3113aa25e56877145c05d4471fb1b3bddc"}
Nov 03 06:08:41.049817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.048401    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" event={"ID":"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3","Type":"ContainerStarted","Data":"32feec40a3f812ad1c84b6a3abf5d3d5a1f9cfc57049547dec43978445e092d7"}
Nov 03 06:08:41.060817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.053204    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" event={"ID":"80326b77-b409-4f1a-9deb-145840baaa82","Type":"ContainerStarted","Data":"ecb23241f9bbcd9cda15f66158fa5d1d8c308e7db5451e723d21a00efd0362be"}
Nov 03 06:08:41.060817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.053256    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" event={"ID":"80326b77-b409-4f1a-9deb-145840baaa82","Type":"ContainerStarted","Data":"a07396e578de25b4ac4a404d387598501e499c361c02ff33af9be8273ebf5ea0"}
Nov 03 06:08:41.110824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.107087    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podStartSLOduration=2.107062403 podStartE2EDuration="2.107062403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:41.099138853 +0000 UTC m=+809.510535932" watchObservedRunningTime="2026-11-03 06:08:41.107062403 +0000 UTC m=+809.518459483"
Nov 03 06:08:41.129813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.125759    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:41.129813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.125840    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:41.157042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.156964    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podStartSLOduration=2.156943855 podStartE2EDuration="2.156943855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:41.153363708 +0000 UTC m=+809.564760788" watchObservedRunningTime="2026-11-03 06:08:41.156943855 +0000 UTC m=+809.568340934"
Nov 03 06:08:41.841247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.839517    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:41.854085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.851918    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:41.854085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.851993    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:41.915634 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.910778    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:41.929857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.925951    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:41.929857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:41.926033    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:42.129877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.129313    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:42.129877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.129392    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:42.843937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.841235    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:42.843937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.841299    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:42.922819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.912872    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:42.922819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:42.912940    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:43.134878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.134344    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:43.134878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.134405    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:43.813618 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:08:43.813576    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:08:43.844059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.844022    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:43.844567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.844538    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:43.948411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.931974    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:43.948411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:43.932058    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:44.124279 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.124183    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:44.125804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.125753    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:44.840195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.840146    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:44.840854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.840822    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:44.906818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.906201    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:44.906818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:44.906267    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:45.126870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.126202    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:45.126870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.126264    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:45.850829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.850372    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:45.850829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.850448    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:45.911105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.910343    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:45.911105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:45.910407    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:46.118871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.117761    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.134819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.124145    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:46.134819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.124230    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:46.176260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.175007    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.221419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.221367    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.308951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.308898    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.311602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.311565    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.359978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.359929    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.466992 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.466888    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:08:46.630472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.630420    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:46.838950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.838908    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:46.839635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.839603    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:46.908817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.908750    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:46.909334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:46.908835    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:47.004817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.004244    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:47.148995 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.148889    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:47.177019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.176969    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:47.198818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.187072    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:08:47.198818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.187152    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:08:47.232818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.207932    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:47.233222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.233141    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:08:47.233443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.233418    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:48.100934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.258909    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:08:48.158827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.366038    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.184433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.159206    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:08:48.184433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.160111    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.184433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.483553    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.184433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.483488    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.201051    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.201132    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.203318    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.203386    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:47.368876    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.343127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.243222    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:48.435876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.409147    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.556711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.427159    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.596643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.450212    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.597017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.452873    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:48.597151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.479554    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:08:48.597291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.597263    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.600806 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.600746    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.765241 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.765154    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:08:48.765480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.765453    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:48.793854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.792928    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:08:48.871728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.871689    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:48.872037 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.872002    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:49.012952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:48.644602    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:49.070661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.070617    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:49.089819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.076287    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:49.089819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.076353    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:49.090531 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:08:49.090498    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.038s"
Nov 03 06:08:49.090823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.090785    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:08:49.092010 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.091988    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:08:49.092941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.092926    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:08:49.096898 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.096881    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:08:49.097617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.097604    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:08:49.125382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.125343    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:49.125991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.125959    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:49.184954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.184907    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:49.185559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.185526    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:08:49.366865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.346046    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"]
Nov 03 06:08:49.431110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.431077    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.461864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.461327    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:08:49.461864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.461481    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:08:49.461864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.461547    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:08:49.461864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.461634    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:08:49.501392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.501362    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:08:49.516824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.512451    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:08:49.516824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.514110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.516824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.514208    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.516824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.514252    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8kjp\" (UniqueName: \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.571817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.535027    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:08:49.650981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.610871    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:08:49.650981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.612530    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:49.653849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.653811    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.662217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.662194    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.662390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.662367    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8kjp\" (UniqueName: \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.665545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.665518    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:49.668959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.668455    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"]
Nov 03 06:08:49.854836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.816563    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:49.854836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.853919    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:49.854836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.854023    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:08:49.854836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.854364    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:08:49.880677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.860067    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:49.880677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.860111    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:49.935044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.934452    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:49.935044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.934517    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:49.942821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.938068    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:08:49.974213 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.974092    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:08:49.980119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.980092    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:08:50.019817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:49.997998    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:08:50.054339 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.054304    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:50.064138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.063380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8kjp\" (UniqueName: \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\") pod \"maas-api-revoke-sub-3f6a8d76-hmwhs\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:50.069240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.069214    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:08:50.070894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.070873    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:08:50.079641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.079612    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:50.123509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.123469    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body=
Nov 03 06:08:50.123758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.123729    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused"
Nov 03 06:08:50.488087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.488006    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"]
Nov 03 06:08:50.783884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.783758    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs" event={"ID":"898d8715-d64b-42d8-b573-b2b7f9ba9756","Type":"ContainerStarted","Data":"0356d9189e1017cbdf901eb087fdf5e3b710edee2cbda4337c40f685b01c717d"}
Nov 03 06:08:50.838983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.838939    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:08:50.839187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.839023    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:08:50.904700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.904626    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:08:50.904891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:50.904686    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:08:51.361817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.361096    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:51.362922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.362429    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:51.363498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.363479    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:08:51.825378 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.825338    4528 generic.go:356] "Generic (PLEG): container finished" podID="898d8715-d64b-42d8-b573-b2b7f9ba9756" containerID="a02484401a183e003229fa57920333a5b689d699d8b8e31b769c5ad9afb2487c" exitCode=0
Nov 03 06:08:51.826850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.826819    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs" event={"ID":"898d8715-d64b-42d8-b573-b2b7f9ba9756","Type":"ContainerDied","Data":"a02484401a183e003229fa57920333a5b689d699d8b8e31b769c5ad9afb2487c"}
Nov 03 06:08:51.846404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.845131    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:51.846640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.846533    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:51.848370 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.847424    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:08:51.909126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.909044    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:51.909501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.909477    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:51.910364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:51.910339    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:08:53.034697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.034661    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:53.130226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.130140    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\") pod \"898d8715-d64b-42d8-b573-b2b7f9ba9756\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") "
Nov 03 06:08:53.130226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.130222    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\" (UniqueName: \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\") pod \"898d8715-d64b-42d8-b573-b2b7f9ba9756\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") "
Nov 03 06:08:53.130404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.130325    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\") pod \"898d8715-d64b-42d8-b573-b2b7f9ba9756\" (UID: \"898d8715-d64b-42d8-b573-b2b7f9ba9756\") "
Nov 03 06:08:53.130531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.130509    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca" pod "898d8715-d64b-42d8-b573-b2b7f9ba9756" (UID: "898d8715-d64b-42d8-b573-b2b7f9ba9756"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:08:53.130956 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.130931    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:53.132337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.132309    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca" pod "898d8715-d64b-42d8-b573-b2b7f9ba9756" (UID: "898d8715-d64b-42d8-b573-b2b7f9ba9756"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:08:53.132455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.132430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp" pod "898d8715-d64b-42d8-b573-b2b7f9ba9756" (UID: "898d8715-d64b-42d8-b573-b2b7f9ba9756"). InnerVolumeSpecName "kube-api-access-r8kjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:08:53.232257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.232217    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8kjp\" (UniqueName: \"kubernetes.io/projected/898d8715-d64b-42d8-b573-b2b7f9ba9756-kube-api-access-r8kjp\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:53.232257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.232252    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/898d8715-d64b-42d8-b573-b2b7f9ba9756-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:53.829948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.829914    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:08:53.830644 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.830624    4528 cpu_manager.go:414] "containerMap: removing container" podUID="898d8715-d64b-42d8-b573-b2b7f9ba9756" containerName="revoke-keys"
Nov 03 06:08:53.830745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.830645    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="898d8715-d64b-42d8-b573-b2b7f9ba9756" containerName="revoke-keys"
Nov 03 06:08:53.830869 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.830856    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="898d8715-d64b-42d8-b573-b2b7f9ba9756" containerName="revoke-keys"
Nov 03 06:08:53.834702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.834680    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.857515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.857454    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs" event={"ID":"898d8715-d64b-42d8-b573-b2b7f9ba9756","Type":"ContainerDied","Data":"0356d9189e1017cbdf901eb087fdf5e3b710edee2cbda4337c40f685b01c717d"}
Nov 03 06:08:53.857723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.857696    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"
Nov 03 06:08:53.858471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.857719    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0356d9189e1017cbdf901eb087fdf5e3b710edee2cbda4337c40f685b01c717d"
Nov 03 06:08:53.858681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.858484    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:08:53.943163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943126    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943171    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943543    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943649 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943595    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943731    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943784 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943765    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.943933 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.943909    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.944068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.944054    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:53.944121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:53.944110    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rflbm\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.045598 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045503    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.045598 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045557    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rflbm\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.045598 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045599    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045622    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045701    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045772    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045813    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.045895    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.046040    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.046352    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.046442    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.046509    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.046955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.046933    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.048115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.048096    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.048710 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.048690    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.069599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.069570    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.069757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.069603    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rflbm\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\") pod \"e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") " pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.159832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.159784    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:54.483110 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:54.483075    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5db0f042_5cdf_43a2_a56e_a44116bd2e04.slice/crio-35acc0515d6a6299a09ebc372debd18efb16b91a40c3c43f5f223f70ef9b7eba WatchSource:0}: Error finding container 35acc0515d6a6299a09ebc372debd18efb16b91a40c3c43f5f223f70ef9b7eba: Status 404 returned error can't find the container with id 35acc0515d6a6299a09ebc372debd18efb16b91a40c3c43f5f223f70ef9b7eba
Nov 03 06:08:54.484474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.484448    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:08:54.486603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.486566    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:54.486688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.486635    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:54.486688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.486666    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:54.572675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.572647    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:08:54.577380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.577351    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.602402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.602369    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:08:54.663727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.663680    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.663982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.663751    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.663982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.663786    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.663982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.663865    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.665573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.664890    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.665573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.664932    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.665573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.665012    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvsgh\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.665573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.665135    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.665573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.665162    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.766694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.766605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.766875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.766750    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.766875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.766783    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.766875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.766867    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvsgh\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767008    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767039    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767092    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767139    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767152    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767169    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767467 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767441    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.767594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.767528    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.768153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.768128    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.769746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.769723    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.770787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.770756    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.790598 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.790564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.791455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.791427    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvsgh\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\") pod \"e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") " pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.870205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.870162    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" event={"ID":"5db0f042-5cdf-43a2-a56e-a44116bd2e04","Type":"ContainerStarted","Data":"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"}
Nov 03 06:08:54.870373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.870225    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" event={"ID":"5db0f042-5cdf-43a2-a56e-a44116bd2e04","Type":"ContainerStarted","Data":"35acc0515d6a6299a09ebc372debd18efb16b91a40c3c43f5f223f70ef9b7eba"}
Nov 03 06:08:54.899760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.899721    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:54.924628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:54.924576    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podStartSLOduration=1.9245624989999999 podStartE2EDuration="1.924562499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:54.924002551 +0000 UTC m=+823.335399629" watchObservedRunningTime="2026-11-03 06:08:54.924562499 +0000 UTC m=+823.335959577"
Nov 03 06:08:55.160705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.160665    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:08:55.162062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.162039    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:08:55.162163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.162086    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:08:55.238360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.238326    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:08:55.240108 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:08:55.240081    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f64805_2cbe_4e03_8f34_2104a7f76cd0.slice/crio-7321266670ef7bfb28243d516f224c7bbc52d94b665f0308155dc2466a3a5de8 WatchSource:0}: Error finding container 7321266670ef7bfb28243d516f224c7bbc52d94b665f0308155dc2466a3a5de8: Status 404 returned error can't find the container with id 7321266670ef7bfb28243d516f224c7bbc52d94b665f0308155dc2466a3a5de8
Nov 03 06:08:55.242875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.242841    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:55.242960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.242909    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:55.242960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.242940    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:08:55.830916 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.830787    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"]
Nov 03 06:08:55.838815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.835782    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:55.918904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.902735    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:08:55.943839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.929341    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:08:55.943839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.930259    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:08:55.943839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.930986    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:08:55.960850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.954101    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"]
Nov 03 06:08:55.960850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.959264    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:55.960850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.959492    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:55.960850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.959568    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzffd\" (UniqueName: \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:55.974605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.974572    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" event={"ID":"d0f64805-2cbe-4e03-8f34-2104a7f76cd0","Type":"ContainerStarted","Data":"b00103f56f6fa98af108eede2c7aebb79f4f4cef993c0b66b8b0513a922791d9"}
Nov 03 06:08:55.974878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:55.974855    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" event={"ID":"d0f64805-2cbe-4e03-8f34-2104a7f76cd0","Type":"ContainerStarted","Data":"7321266670ef7bfb28243d516f224c7bbc52d94b665f0308155dc2466a3a5de8"}
Nov 03 06:08:56.062127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.062081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.062785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.062759    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.062899 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.062867    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzffd\" (UniqueName: \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.063658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.063628    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.066416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.066394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.094661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.094579    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzffd\" (UniqueName: \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\") pod \"maas-api-revoke-sub-b2c43dd0-jbg8c\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.160171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.160139    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:08:56.160327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.160202    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:08:56.268283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.268248    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:56.755588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.754736    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podStartSLOduration=2.754717202 podStartE2EDuration="2.754717202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:08:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:08:56.035187376 +0000 UTC m=+824.446584456" watchObservedRunningTime="2026-11-03 06:08:56.754717202 +0000 UTC m=+825.166114282"
Nov 03 06:08:56.759164 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.759123    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"]
Nov 03 06:08:56.903561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.903528    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:08:56.904000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.903976    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:08:56.904214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:56.904190    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:08:57.017547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:57.017450    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c" event={"ID":"7f391aab-116a-44f6-b17c-55ebe692c8d6","Type":"ContainerStarted","Data":"2a51895dc08919b17bbafbe85eadda4e1f313bded57fc4bcbbb6b5b2472abce3"}
Nov 03 06:08:57.160261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:57.160222    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:08:57.160438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:57.160270    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:08:57.940026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:57.939967    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:08:57.940525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:57.940041    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:08:58.055234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.055198    4528 generic.go:356] "Generic (PLEG): container finished" podID="7f391aab-116a-44f6-b17c-55ebe692c8d6" containerID="a1826aa26b6febd799163afe76cffbc071da9c30ccf007b21946c7f1ceb53101" exitCode=0
Nov 03 06:08:58.055404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.055322    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c" event={"ID":"7f391aab-116a-44f6-b17c-55ebe692c8d6","Type":"ContainerDied","Data":"a1826aa26b6febd799163afe76cffbc071da9c30ccf007b21946c7f1ceb53101"}
Nov 03 06:08:58.161027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.160998    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:08:58.161241 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.161221    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:08:58.905815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.905301    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:08:58.905815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:58.905366    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:08:59.161329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.161250    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:08:59.161329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.161305    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:08:59.451291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.451220    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:08:59.455555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.455532    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\") pod \"7f391aab-116a-44f6-b17c-55ebe692c8d6\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") "
Nov 03 06:08:59.455655 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.455612    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\") pod \"7f391aab-116a-44f6-b17c-55ebe692c8d6\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") "
Nov 03 06:08:59.455655 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.455647    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\" (UniqueName: \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\") pod \"7f391aab-116a-44f6-b17c-55ebe692c8d6\" (UID: \"7f391aab-116a-44f6-b17c-55ebe692c8d6\") "
Nov 03 06:08:59.456003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.455975    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca" pod "7f391aab-116a-44f6-b17c-55ebe692c8d6" (UID: "7f391aab-116a-44f6-b17c-55ebe692c8d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:08:59.456155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.456141    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:59.459023 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.458997    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca" pod "7f391aab-116a-44f6-b17c-55ebe692c8d6" (UID: "7f391aab-116a-44f6-b17c-55ebe692c8d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:08:59.459257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.459229    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd" pod "7f391aab-116a-44f6-b17c-55ebe692c8d6" (UID: "7f391aab-116a-44f6-b17c-55ebe692c8d6"). InnerVolumeSpecName "kube-api-access-bzffd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:08:59.566581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.566539    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzffd\" (UniqueName: \"kubernetes.io/projected/7f391aab-116a-44f6-b17c-55ebe692c8d6-kube-api-access-bzffd\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:59.566881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.566787    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f391aab-116a-44f6-b17c-55ebe692c8d6-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:08:59.911362 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.911321    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:08:59.911665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:08:59.911634    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:09:00.141053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.141010    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c" event={"ID":"7f391aab-116a-44f6-b17c-55ebe692c8d6","Type":"ContainerDied","Data":"2a51895dc08919b17bbafbe85eadda4e1f313bded57fc4bcbbb6b5b2472abce3"}
Nov 03 06:09:00.142842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.142819    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a51895dc08919b17bbafbe85eadda4e1f313bded57fc4bcbbb6b5b2472abce3"
Nov 03 06:09:00.142989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.142736    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"
Nov 03 06:09:00.160339 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.160306    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:09:00.160659 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.160605    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:09:00.903304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.903267    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:09:00.904041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:00.903996    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:09:01.161111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:01.161021    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body=
Nov 03 06:09:01.161111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:01.161082    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused"
Nov 03 06:09:01.905657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:01.904571    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused" start-of-body=
Nov 03 06:09:01.905657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:01.904650    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": dial tcp 10.132.0.112:15021: connect: connection refused"
Nov 03 06:09:02.194190 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.193279    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:09:02.195710 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.194666    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:09:02.195851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.195777    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:09:02.382852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.382808    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"]
Nov 03 06:09:02.383873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.383851    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7f391aab-116a-44f6-b17c-55ebe692c8d6" containerName="revoke-keys"
Nov 03 06:09:02.383962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.383875    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f391aab-116a-44f6-b17c-55ebe692c8d6" containerName="revoke-keys"
Nov 03 06:09:02.384079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.384067    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f391aab-116a-44f6-b17c-55ebe692c8d6" containerName="revoke-keys"
Nov 03 06:09:02.388976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.388955    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.408271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.408238    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:02.408466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.408434    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:02.427886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.408591    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:02.427886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.408251    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:02.445369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.445274    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.445551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.445374    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.445551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.445493    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7rjs\" (UniqueName: \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.447251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.447199    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"]
Nov 03 06:09:02.546663 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.546620    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.546896 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.546749    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7rjs\" (UniqueName: \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.547036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.547018    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.547838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.547809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.550142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.550111    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.570208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.570166    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7rjs\" (UniqueName: \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\") pod \"maas-api-revoke-sub-9d834b2b-72hvq\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.758381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.758291    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:02.957363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.957327    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:09:02.958412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.958383    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:09:02.959186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:02.959167    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:09:03.093474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:03.093427    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"]
Nov 03 06:09:03.178203 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:03.178169    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq" event={"ID":"3420df3c-9273-4979-9a34-42e329a21d1c","Type":"ContainerStarted","Data":"03151c97eb51b8c3918df8c1accded0f6f449c589fad5263e805d4cbc7506ec6"}
Nov 03 06:09:04.190379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:04.190330    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq" event={"ID":"3420df3c-9273-4979-9a34-42e329a21d1c","Type":"ContainerStarted","Data":"c63650f08bad0358da32b86277e52911aa4f55758fe0d824022d3f436f67bc30"}
Nov 03 06:09:04.230228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:04.230175    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq" podStartSLOduration=2.230160004 podStartE2EDuration="2.230160004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:04.228861399 +0000 UTC m=+832.640258478" watchObservedRunningTime="2026-11-03 06:09:04.230160004 +0000 UTC m=+832.641557082"
Nov 03 06:09:05.203272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:05.203236    4528 generic.go:356] "Generic (PLEG): container finished" podID="3420df3c-9273-4979-9a34-42e329a21d1c" containerID="c63650f08bad0358da32b86277e52911aa4f55758fe0d824022d3f436f67bc30" exitCode=0
Nov 03 06:09:05.203680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:05.203323    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq" event={"ID":"3420df3c-9273-4979-9a34-42e329a21d1c","Type":"ContainerDied","Data":"c63650f08bad0358da32b86277e52911aa4f55758fe0d824022d3f436f67bc30"}
Nov 03 06:09:06.593205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.593167    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:06.683408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.683372    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\") pod \"3420df3c-9273-4979-9a34-42e329a21d1c\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") "
Nov 03 06:09:06.684166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.684139    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\") pod \"3420df3c-9273-4979-9a34-42e329a21d1c\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") "
Nov 03 06:09:06.684853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.684832    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\" (UniqueName: \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\") pod \"3420df3c-9273-4979-9a34-42e329a21d1c\" (UID: \"3420df3c-9273-4979-9a34-42e329a21d1c\") "
Nov 03 06:09:06.686103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.684658    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca" pod "3420df3c-9273-4979-9a34-42e329a21d1c" (UID: "3420df3c-9273-4979-9a34-42e329a21d1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:06.686407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.686387    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:06.703487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.703449    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca" pod "3420df3c-9273-4979-9a34-42e329a21d1c" (UID: "3420df3c-9273-4979-9a34-42e329a21d1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:06.705773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.705727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs" pod "3420df3c-9273-4979-9a34-42e329a21d1c" (UID: "3420df3c-9273-4979-9a34-42e329a21d1c"). InnerVolumeSpecName "kube-api-access-h7rjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:06.788950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.788882    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3420df3c-9273-4979-9a34-42e329a21d1c-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:06.788950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:06.788922    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7rjs\" (UniqueName: \"kubernetes.io/projected/3420df3c-9273-4979-9a34-42e329a21d1c-kube-api-access-h7rjs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:07.312134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:07.303349    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq" event={"ID":"3420df3c-9273-4979-9a34-42e329a21d1c","Type":"ContainerDied","Data":"03151c97eb51b8c3918df8c1accded0f6f449c589fad5263e805d4cbc7506ec6"}
Nov 03 06:09:07.312134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:07.303418    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03151c97eb51b8c3918df8c1accded0f6f449c589fad5263e805d4cbc7506ec6"
Nov 03 06:09:07.312134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:07.303496    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"
Nov 03 06:09:08.509018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:08.508953    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:08.614965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:08.614916    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:09:08.801298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:08.801165    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:08.858019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:08.857962    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:08.980969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:08.980910    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.109818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.103054    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.253819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.238233    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.417929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.351023    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:09:09.417929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.352164    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3420df3c-9273-4979-9a34-42e329a21d1c" containerName="revoke-keys"
Nov 03 06:09:09.417929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.352198    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3420df3c-9273-4979-9a34-42e329a21d1c" containerName="revoke-keys"
Nov 03 06:09:09.417929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.352416    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3420df3c-9273-4979-9a34-42e329a21d1c" containerName="revoke-keys"
Nov 03 06:09:09.421983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.421915    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.525383    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.650953    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.650998    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.699198    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.699256    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:09.715876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.700289    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:09.782123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.781042    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.865298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.832911    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:09:09.924830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.866866    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:09.991508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:09.991419    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:10.165848 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.104049    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.107782    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zgx\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166231    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166395    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166489    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166592    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.166748    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.206206    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.206285    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.206309    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.206369    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:10.236906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.206401    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:10.260950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.260887    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:10.274979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.274942    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:10.277117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.277081    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.354918    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355005    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355084    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355122    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355306    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355532    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.355808    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zgx\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.357209    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.358011    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.358531    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.359226    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.359530    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.491881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.435908    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"]
Nov 03 06:09:10.597862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.561566    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:10.802882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.799046    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.820812    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.821284    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.822062    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zgx\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\") pod \"e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") " pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.852241    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:10.852286    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.040345    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"]
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.040510    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.125199    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:11.572830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.308300    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:09:11.575880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.315864    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:09:11.576160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.317809    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:11.584247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.317888    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:09:11.584492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.441195    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": context deadline exceeded"
Nov 03 06:09:11.584645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.543592    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:11.588816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.586003    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2s2\" (UniqueName: \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:11.636815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.630191    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:11.636815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.633002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:11.763211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.741677    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:11.943094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.942995    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:11.943586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.943103    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:11.981818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.974125    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:11.983008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:11.982980    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:12.620030 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.604912    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:12.633818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.630229    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:12.968944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.968782    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:12.970099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.970069    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:09:12.971780 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.971754    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:12.975358 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.945205    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:12.998158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:12.998124    4528 patch_prober.go:29] interesting pod/jobset-controller-manager-f8868cd4b-pc7rp container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:13.014908 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.014866    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-f8868cd4b-pc7rp" podUID="c6e0c543-0071-4228-a24b-c0c6c504ea4d" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.42:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:13.015156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.000841    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:13.092955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.070563    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.100384 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.099938    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2s2\" (UniqueName: \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.100384 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.100120    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.103620 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.102156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.115427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.115398    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.119441 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:09:13.119413    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.326s"
Nov 03 06:09:13.119638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.119623    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:09:13.136247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.136211    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:09:13.136548 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.136487    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.137518 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.137441    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:13.150522 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.150490    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2s2\" (UniqueName: \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\") pod \"maas-api-revoke-sub-70051005-z98sh\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.171090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171053    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:09:13.171261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171182    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:13.171261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171250    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:09:13.171383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171278    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:09:13.171383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171352    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:13.171490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171413    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:13.171490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171470    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:13.171599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171526    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:13.171689 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171666    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:13.171826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171787    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:13.171898 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.171857    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:09:13.175223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.175198    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.184871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.184845    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:09:13.185145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.185130    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.200178 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.200138    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"kube-root-ca.crt\""
Nov 03 06:09:13.200421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.200190    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"openshift-service-ca.crt\""
Nov 03 06:09:13.200597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.200495    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"distinct-2-w1-9688d8-kserve-self-signed-certs\""
Nov 03 06:09:13.201038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.201016    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-9688d8\"/\"default-dockercfg-zgwhm\""
Nov 03 06:09:13.281534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.281454    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:13.303935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.303898    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.304079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.303967    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.304079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304039    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.304162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304148    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.304212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304194    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.304252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304224    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.304289 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304267    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.304328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304302    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m2pn\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.304368 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304353    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.304423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304409    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.304466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.304439    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.305460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305432    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.305571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305532    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.305628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305613    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.305712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305697    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7z75\" (UniqueName: \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.305748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305727    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.305842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.305885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305860    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.305979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.305961    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.306031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.306017    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.306072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.306049    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.306129 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.306115    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.306190 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.306178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.306248 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.306227    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkf8\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.335861 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:13.335827    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81219190_ee1a_4ecf_acdf_c29942df3e8c.slice/crio-36659bbadd93d4d1fc9704a8a7689f5208c11d7056f944807604cf01325ed3c8 WatchSource:0}: Error finding container 36659bbadd93d4d1fc9704a8a7689f5208c11d7056f944807604cf01325ed3c8: Status 404 returned error can't find the container with id 36659bbadd93d4d1fc9704a8a7689f5208c11d7056f944807604cf01325ed3c8
Nov 03 06:09:13.337883 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.337858    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:09:13.338845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.338782    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.338968 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.338878    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.338968 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.338917    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.407375 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407341    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407384    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407401    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407420    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407441    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m2pn\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407491    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407511    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.407541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407541    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407562    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407582    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407626    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7z75\" (UniqueName: \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407643    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407681    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407711    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407761    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407810    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.407842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407840    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.408119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407887    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.408119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407931    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.408119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.407984    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkf8\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.408119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.408102    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.408266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.408132    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.408266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.408185    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.410328    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.410408    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.410730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.410994    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411236    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411251    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411342    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411533    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411627    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.411949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.411844    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.412715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.412100    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.412715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.412128    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.412715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.412570    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.415025 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.413504    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.415025 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.414573    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.415157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.415102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.416405 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.416360    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.417536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.417514    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.423244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.423173    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.434086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.434024    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7z75\" (UniqueName: \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\") pod \"distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") " pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.436408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.436384    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.438582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.438556    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m2pn\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.439222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.439168    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.440567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.440544    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkf8\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\") pod \"e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") " pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.485200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.485151    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:13.500186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.500139    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:13.512877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.512844    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:13.625294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.625252    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"]
Nov 03 06:09:13.732013 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:09:13.731063    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:09:13.883421 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:13.883393    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba872758_9e9a_451f_b3c7_1e9563c31336.slice/crio-61aa9649ae569219a53fb9fe2d8f281f73e7626401fad255b3c48bcb4a9d358f WatchSource:0}: Error finding container 61aa9649ae569219a53fb9fe2d8f281f73e7626401fad255b3c48bcb4a9d358f: Status 404 returned error can't find the container with id 61aa9649ae569219a53fb9fe2d8f281f73e7626401fad255b3c48bcb4a9d358f
Nov 03 06:09:13.888788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.888752    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:09:13.896906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.896864    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.897007 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.896956    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.897058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.897001    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.926044 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:13.924444    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18544b11_2328_4148_b9a8_2533f1fc4822.slice/crio-2787f492276f7f7f8b0f1b16484bb4b849a082991af7d0cf969d534a2c613018 WatchSource:0}: Error finding container 2787f492276f7f7f8b0f1b16484bb4b849a082991af7d0cf969d534a2c613018: Status 404 returned error can't find the container with id 2787f492276f7f7f8b0f1b16484bb4b849a082991af7d0cf969d534a2c613018
Nov 03 06:09:13.927559 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:13.927529    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc309c4d0_3513_4c7b_991d_88c891e6ce54.slice/crio-78feb78cabcfe33ca3e18b41ea1f6af9e51ca5e13b4fb56ec39f71232eb57df7 WatchSource:0}: Error finding container 78feb78cabcfe33ca3e18b41ea1f6af9e51ca5e13b4fb56ec39f71232eb57df7: Status 404 returned error can't find the container with id 78feb78cabcfe33ca3e18b41ea1f6af9e51ca5e13b4fb56ec39f71232eb57df7
Nov 03 06:09:13.932623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.932329    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.932623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.932427    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.932623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.932475    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:13.942230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.939588    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:09:13.946627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:13.946591    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:09:14.137680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.137597    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" event={"ID":"ba872758-9e9a-451f-b3c7-1e9563c31336","Type":"ContainerStarted","Data":"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"}
Nov 03 06:09:14.138263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.138229    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" event={"ID":"ba872758-9e9a-451f-b3c7-1e9563c31336","Type":"ContainerStarted","Data":"61aa9649ae569219a53fb9fe2d8f281f73e7626401fad255b3c48bcb4a9d358f"}
Nov 03 06:09:14.140317 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.140287    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" event={"ID":"18544b11-2328-4148-b9a8-2533f1fc4822","Type":"ContainerStarted","Data":"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"}
Nov 03 06:09:14.140452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.140341    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" event={"ID":"18544b11-2328-4148-b9a8-2533f1fc4822","Type":"ContainerStarted","Data":"2787f492276f7f7f8b0f1b16484bb4b849a082991af7d0cf969d534a2c613018"}
Nov 03 06:09:14.142282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.142257    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" event={"ID":"c309c4d0-3513-4c7b-991d-88c891e6ce54","Type":"ContainerStarted","Data":"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"}
Nov 03 06:09:14.142397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.142293    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" event={"ID":"c309c4d0-3513-4c7b-991d-88c891e6ce54","Type":"ContainerStarted","Data":"78feb78cabcfe33ca3e18b41ea1f6af9e51ca5e13b4fb56ec39f71232eb57df7"}
Nov 03 06:09:14.144388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.144366    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" event={"ID":"81219190-ee1a-4ecf-acdf-c29942df3e8c","Type":"ContainerStarted","Data":"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"}
Nov 03 06:09:14.144487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.144406    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" event={"ID":"81219190-ee1a-4ecf-acdf-c29942df3e8c","Type":"ContainerStarted","Data":"36659bbadd93d4d1fc9704a8a7689f5208c11d7056f944807604cf01325ed3c8"}
Nov 03 06:09:14.146257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.146234    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh" event={"ID":"e000c21d-1a21-423f-b288-fddb0c903536","Type":"ContainerStarted","Data":"a955169d18651bb5048e74197e608fe6202d4b7440eb4ac18316b83f39bd5818"}
Nov 03 06:09:14.220918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.220780    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podStartSLOduration=5.220758529 podStartE2EDuration="5.220758529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:14.18159047 +0000 UTC m=+842.592987548" watchObservedRunningTime="2026-11-03 06:09:14.220758529 +0000 UTC m=+842.632155608"
Nov 03 06:09:14.235335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.235281    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podStartSLOduration=2.235268503 podStartE2EDuration="2.235268503s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:14.225348423 +0000 UTC m=+842.636745500" watchObservedRunningTime="2026-11-03 06:09:14.235268503 +0000 UTC m=+842.646665580"
Nov 03 06:09:14.273221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.273161    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podStartSLOduration=5.273146574 podStartE2EDuration="5.273146574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:14.268560202 +0000 UTC m=+842.679957280" watchObservedRunningTime="2026-11-03 06:09:14.273146574 +0000 UTC m=+842.684543657"
Nov 03 06:09:14.332380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.332301    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podStartSLOduration=6.332276247 podStartE2EDuration="6.332276247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:14.311989645 +0000 UTC m=+842.723386765" watchObservedRunningTime="2026-11-03 06:09:14.332276247 +0000 UTC m=+842.743673326"
Nov 03 06:09:14.500879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.497162    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:14.501051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.500891    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:14.503124 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.503093    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:14.503333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.503309    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:14.514935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.514899    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:14.630923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.630778    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:14.632381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.632345    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:14.632513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:14.632401    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:15.175168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.175073    4528 generic.go:356] "Generic (PLEG): container finished" podID="e000c21d-1a21-423f-b288-fddb0c903536" containerID="da2c57d13ecae4815902de36e6ae943c9d932481676c813f9b8ae998b5bc47a3" exitCode=0
Nov 03 06:09:15.200263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.184966    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh" event={"ID":"e000c21d-1a21-423f-b288-fddb0c903536","Type":"ContainerDied","Data":"da2c57d13ecae4815902de36e6ae943c9d932481676c813f9b8ae998b5bc47a3"}
Nov 03 06:09:15.531067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.530961    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:15.627240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.627174    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:15.662818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.658570    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:15.662818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.662161    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:15.662818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.662343    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:15.662818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:15.662375    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:16.506557 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.506516    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:16.519568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.517885    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:16.519568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.510899    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:16.519568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.518302    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:16.586210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.586142    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:16.673838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.673751    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:16.674055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.673845    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:16.903832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:16.891143    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:17.049098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.049049    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:17.142902 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.142839    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:17.518768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.518713    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 03 06:09:17.551833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.551586    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:17.551833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.551655    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:17.585817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.558399    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:17.585817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.558467    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:17.607814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.607118    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:17.607814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.607652    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:09:17.608424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.608184    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:17.707465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.665070    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:17.707465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.665155    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:17.924767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.915146    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:09:17.924767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:17.918350    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:18.052246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.051953    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:18.089844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.060199    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:18.180210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.180088    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.644210    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.644276    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.645310    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.645363    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.648690    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:18.648817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.648754    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:18.649592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.648879    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:18.649592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.648903    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:18.652288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.652240    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:18.654591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.652476    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:18.655373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.655355    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:18.818822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.801914    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:18.991505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:18.991407    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:19.068107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.052883    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:19.085818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.076510    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:09:19.085818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.076963    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:19.245895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.230902    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:19.465704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.462033    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Nov 03 06:09:19.521113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.487366    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:19.521113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.487422    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:19.521776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.521743    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:19.522009 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.521975    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:19.669629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.669589    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:19.672491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.672454    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:19.725816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.723378    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:19.861815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:19.816881    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": context deadline exceeded"
Nov 03 06:09:20.016241 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.016024    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.016769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.016749    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:20.037278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.015965    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.058015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.026881    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.078705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.078655    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.100085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.100038    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.112882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.108947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\" (UniqueName: \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\") pod \"e000c21d-1a21-423f-b288-fddb0c903536\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") "
Nov 03 06:09:20.145815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.121068    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\") pod \"e000c21d-1a21-423f-b288-fddb0c903536\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") "
Nov 03 06:09:20.248875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.227702    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\") pod \"e000c21d-1a21-423f-b288-fddb0c903536\" (UID: \"e000c21d-1a21-423f-b288-fddb0c903536\") "
Nov 03 06:09:20.276360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.263540    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:20.338293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.264240    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca" pod "e000c21d-1a21-423f-b288-fddb0c903536" (UID: "e000c21d-1a21-423f-b288-fddb0c903536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:20.338537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.264304    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.436930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.420956    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:20.437114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.437024    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca" pod "e000c21d-1a21-423f-b288-fddb0c903536" (UID: "e000c21d-1a21-423f-b288-fddb0c903536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:20.553596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.553545    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2" pod "e000c21d-1a21-423f-b288-fddb0c903536" (UID: "e000c21d-1a21-423f-b288-fddb0c903536"). InnerVolumeSpecName "kube-api-access-fm2s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:20.554374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.554349    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e000c21d-1a21-423f-b288-fddb0c903536-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:20.554525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.554510    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2s2\" (UniqueName: \"kubernetes.io/projected/e000c21d-1a21-423f-b288-fddb0c903536-kube-api-access-fm2s2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:20.555514 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.555486    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:20.555713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.555678    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:20.594152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.594099    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:20.594448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.594396    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:20.722910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.720146    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:09:20.767003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.766961    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 06:09:20.828824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.793086    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:20.828824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.793148    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:20.850136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.850083    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh" event={"ID":"e000c21d-1a21-423f-b288-fddb0c903536","Type":"ContainerDied","Data":"a955169d18651bb5048e74197e608fe6202d4b7440eb4ac18316b83f39bd5818"}
Nov 03 06:09:20.857819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.850376    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a955169d18651bb5048e74197e608fe6202d4b7440eb4ac18316b83f39bd5818"
Nov 03 06:09:20.860314 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.860292    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"
Nov 03 06:09:20.899818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.895121    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:20.982866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:20.982241    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:21.071777 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.045483    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:09:21.072720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.072689    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:21.114682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.114640    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:09:21.118134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.118099    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:21.169867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.169833    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"]
Nov 03 06:09:21.172758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.172709    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e000c21d-1a21-423f-b288-fddb0c903536" containerName="revoke-keys"
Nov 03 06:09:21.172926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.172911    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e000c21d-1a21-423f-b288-fddb0c903536" containerName="revoke-keys"
Nov 03 06:09:21.173454 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.173439    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e000c21d-1a21-423f-b288-fddb0c903536" containerName="revoke-keys"
Nov 03 06:09:21.273763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.273679    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:21.304857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.301223    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:21.304857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.301277    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:21.304857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.274071    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.336951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.329165    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:21.336951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.333776    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:21.336951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.333905    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:21.336951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.333980    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:21.341242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.341198    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:21.369354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.359102    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"]
Nov 03 06:09:21.421244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.421015    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.421244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.421238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.421469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.421276    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2tzc\" (UniqueName: \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.518816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.500740    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:21.518816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.500818    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:21.518816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.501390    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:21.518816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.501425    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:21.526112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.525679    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.526335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.526301    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2tzc\" (UniqueName: \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.526692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.526671    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.529743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.529165    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.589085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.552885    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2tzc\" (UniqueName: \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.589936 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.589871    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-622f06ef-7bksc\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:21.632256 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.632205    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body=
Nov 03 06:09:21.686305 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.686256    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused"
Nov 03 06:09:21.732658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:21.732607    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:22.106878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.100978    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"]
Nov 03 06:09:22.485613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.485526    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body=
Nov 03 06:09:22.485613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.485592    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused"
Nov 03 06:09:22.502051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.502014    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body=
Nov 03 06:09:22.502199 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.502073    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused"
Nov 03 06:09:22.636874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.636643    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:22.637838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.637775    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:22.639443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.639173    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:09:22.914928 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.913738    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc" event={"ID":"1597ac30-419a-47f2-8802-15fa12ac07f9","Type":"ContainerStarted","Data":"2aae61c0dea394082fbc40faa7d0e8e25be632a70831dc2c8fc34622e727b9a1"}
Nov 03 06:09:22.914928 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.913815    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc" event={"ID":"1597ac30-419a-47f2-8802-15fa12ac07f9","Type":"ContainerStarted","Data":"72013243df03754528ec8497c2c708644a8e3c59f28a05cd70ffdfb5dac364a6"}
Nov 03 06:09:22.955767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:22.955630    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc" podStartSLOduration=2.955609072 podStartE2EDuration="2.955609072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:22.952699511 +0000 UTC m=+851.364096590" watchObservedRunningTime="2026-11-03 06:09:22.955609072 +0000 UTC m=+851.367006150"
Nov 03 06:09:23.148886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.148848    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"]
Nov 03 06:09:23.155278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.155251    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.221494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.221339    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"]
Nov 03 06:09:23.300153 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.300113    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktk6b\" (UniqueName: \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.300363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.300341    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.300434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.300386    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.401870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.401833    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.402028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.401906    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.402112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.402075    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktk6b\" (UniqueName: \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.402889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.402857    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.404621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.404593    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.425764 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.425701    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktk6b\" (UniqueName: \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\") pod \"maas-api-revoke-sub-183fae31-x2n87\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.490633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.490597    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:23.491175 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.491155    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:23.491972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.491951    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:09:23.503392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.503365    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:23.505167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.505148    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:23.505526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.505512    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:23.506176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.506158    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:09:23.513762 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.513738    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:23.527158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.527139    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:23.823997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.823959    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"]
Nov 03 06:09:23.924019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.923979    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87" event={"ID":"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b","Type":"ContainerStarted","Data":"90d5cc600f73ab831a7d786d617689b1874d7d77e53845e460a7cb0b80fcbcc0"}
Nov 03 06:09:23.925723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.925700    4528 generic.go:356] "Generic (PLEG): container finished" podID="1597ac30-419a-47f2-8802-15fa12ac07f9" containerID="2aae61c0dea394082fbc40faa7d0e8e25be632a70831dc2c8fc34622e727b9a1" exitCode=0
Nov 03 06:09:23.925875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.925843    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc" event={"ID":"1597ac30-419a-47f2-8802-15fa12ac07f9","Type":"ContainerDied","Data":"2aae61c0dea394082fbc40faa7d0e8e25be632a70831dc2c8fc34622e727b9a1"}
Nov 03 06:09:23.939446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:23.939419    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:24.376615 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.376539    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:09:24.381993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.381967    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.425027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.423913    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:09:24.522140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522101    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.522490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522464    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhxks\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.522720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522693    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.522876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522832    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.522977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522957    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.523040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.522992    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.523040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.523026    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.523147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.523130    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.523219 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.523201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.613117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.613073    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:09:24.619137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.619103    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.626034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.625999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626099    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhxks\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626186    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626226    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626321    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626350    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626382    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626433    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626445    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626479    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626809    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.626957 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.626891    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.627187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.627162    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.627684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.627636    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.631457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.631432    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.632437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.632404    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.643390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.643364    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:09:24.657875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.653903    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.657875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.654827    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhxks\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.727713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.727670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.727954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.727924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b79wv\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728076    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728115    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728399    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728445    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728572    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728636    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.728703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.728671    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.737547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.737525    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:24.830212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830162    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830245    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830276    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830321    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830464    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b79wv\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830591    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830680    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830826    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.830887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.830881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.831337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.831312    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.831571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.831550    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.831852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.831779    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.832088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.832060    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.832652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.832631    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.834683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.834659    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.836170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.836151    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.854042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.854011    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b79wv\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.854184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.854141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:24.940236 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.940197    4528 generic.go:356] "Generic (PLEG): container finished" podID="4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" containerID="0e37e03ebd46fbeb779f90187a78955497fd07ab682bc852bd023e23c0b0b13d" exitCode=0
Nov 03 06:09:24.940624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.940268    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87" event={"ID":"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b","Type":"ContainerDied","Data":"0e37e03ebd46fbeb779f90187a78955497fd07ab682bc852bd023e23c0b0b13d"}
Nov 03 06:09:24.946390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:24.946366    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:25.088098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.087370    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.088098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.087463    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.088098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.087512    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.092298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.092273    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:09:25.126886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.126858    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:25.136255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.136187    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\") pod \"1597ac30-419a-47f2-8802-15fa12ac07f9\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") "
Nov 03 06:09:25.136408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.136390    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\") pod \"1597ac30-419a-47f2-8802-15fa12ac07f9\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") "
Nov 03 06:09:25.136464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.136449    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\" (UniqueName: \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\") pod \"1597ac30-419a-47f2-8802-15fa12ac07f9\" (UID: \"1597ac30-419a-47f2-8802-15fa12ac07f9\") "
Nov 03 06:09:25.136611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.136580    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca" pod "1597ac30-419a-47f2-8802-15fa12ac07f9" (UID: "1597ac30-419a-47f2-8802-15fa12ac07f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:25.137196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.137172    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:25.138753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.138727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc" pod "1597ac30-419a-47f2-8802-15fa12ac07f9" (UID: "1597ac30-419a-47f2-8802-15fa12ac07f9"). InnerVolumeSpecName "kube-api-access-f2tzc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:25.139901 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.139875    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca" pod "1597ac30-419a-47f2-8802-15fa12ac07f9" (UID: "1597ac30-419a-47f2-8802-15fa12ac07f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:25.238657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.238604    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2tzc\" (UniqueName: \"kubernetes.io/projected/1597ac30-419a-47f2-8802-15fa12ac07f9-kube-api-access-f2tzc\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:25.238657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.238642    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1597ac30-419a-47f2-8802-15fa12ac07f9-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:25.282683 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:25.282650    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b9b9c0_cf19_439f_aaad_6b654d2b2768.slice/crio-464dd19cf2b249fe75f0153b5fe5b2f88786872273e93d44deea28374a5d71a7 WatchSource:0}: Error finding container 464dd19cf2b249fe75f0153b5fe5b2f88786872273e93d44deea28374a5d71a7: Status 404 returned error can't find the container with id 464dd19cf2b249fe75f0153b5fe5b2f88786872273e93d44deea28374a5d71a7
Nov 03 06:09:25.283731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.283702    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:09:25.289329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.289288    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.289429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.289378    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.289429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.289418    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:09:25.953067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.952856    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" event={"ID":"95980fca-5c88-40d0-9004-1609238d9708","Type":"ContainerStarted","Data":"a265c11abb5ae06013da3d718b875c6a20d54bddc711961b3212e81469735e66"}
Nov 03 06:09:25.953067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.952930    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" event={"ID":"95980fca-5c88-40d0-9004-1609238d9708","Type":"ContainerStarted","Data":"b4e75610997274e8fe0977a763828f77c24e29850a370e2b80b7bc214529bbe9"}
Nov 03 06:09:25.955012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.954982    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" event={"ID":"e8b9b9c0-cf19-439f-aaad-6b654d2b2768","Type":"ContainerStarted","Data":"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"}
Nov 03 06:09:25.955130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.955038    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" event={"ID":"e8b9b9c0-cf19-439f-aaad-6b654d2b2768","Type":"ContainerStarted","Data":"464dd19cf2b249fe75f0153b5fe5b2f88786872273e93d44deea28374a5d71a7"}
Nov 03 06:09:25.957476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.957455    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"
Nov 03 06:09:25.957591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.957468    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc" event={"ID":"1597ac30-419a-47f2-8802-15fa12ac07f9","Type":"ContainerDied","Data":"72013243df03754528ec8497c2c708644a8e3c59f28a05cd70ffdfb5dac364a6"}
Nov 03 06:09:25.957591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:25.957508    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72013243df03754528ec8497c2c708644a8e3c59f28a05cd70ffdfb5dac364a6"
Nov 03 06:09:26.036152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.025371    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podStartSLOduration=2.025347561 podStartE2EDuration="2.025347561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:25.999270147 +0000 UTC m=+854.410667227" watchObservedRunningTime="2026-11-03 06:09:26.025347561 +0000 UTC m=+854.436744640"
Nov 03 06:09:26.063546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.063493    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podStartSLOduration=2.063479431 podStartE2EDuration="2.063479431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:26.059810382 +0000 UTC m=+854.471207451" watchObservedRunningTime="2026-11-03 06:09:26.063479431 +0000 UTC m=+854.474876509"
Nov 03 06:09:26.339817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.339007    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:26.358560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.358530    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\") pod \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") "
Nov 03 06:09:26.358715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.358604    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\" (UniqueName: \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\") pod \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") "
Nov 03 06:09:26.358715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.358626    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\") pod \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\" (UID: \"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b\") "
Nov 03 06:09:26.359062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.359034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca" pod "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" (UID: "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:26.359201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.359182    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:26.360765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.360741    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b" pod "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" (UID: "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b"). InnerVolumeSpecName "kube-api-access-ktk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:26.360877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.360820    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca" pod "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" (UID: "4591b4f2-23ed-4282-abdf-ac5e65fe0b2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:26.461410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.461373    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:26.461410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.461412    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktk6b\" (UniqueName: \"kubernetes.io/projected/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b-kube-api-access-ktk6b\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:26.750817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.750467    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:26.751682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.751640    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:26.751956 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.751880    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:26.946969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.946927    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:26.948326 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.948296    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:26.948451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:26.948348    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:27.040855 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.040764    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87" event={"ID":"4591b4f2-23ed-4282-abdf-ac5e65fe0b2b","Type":"ContainerDied","Data":"90d5cc600f73ab831a7d786d617689b1874d7d77e53845e460a7cb0b80fcbcc0"}
Nov 03 06:09:27.041395 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.041377    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90d5cc600f73ab831a7d786d617689b1874d7d77e53845e460a7cb0b80fcbcc0"
Nov 03 06:09:27.041543 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.041533    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"
Nov 03 06:09:27.739921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.739882    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:27.740221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.740192    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:27.947528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.947495    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:27.947728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:27.947708    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:28.035942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.035843    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"]
Nov 03 06:09:28.037443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.037415    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" containerName="revoke-keys"
Nov 03 06:09:28.037620 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.037605    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" containerName="revoke-keys"
Nov 03 06:09:28.037755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.037743    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1597ac30-419a-47f2-8802-15fa12ac07f9" containerName="revoke-keys"
Nov 03 06:09:28.043880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.043469    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1597ac30-419a-47f2-8802-15fa12ac07f9" containerName="revoke-keys"
Nov 03 06:09:28.043880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.043742    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" containerName="revoke-keys"
Nov 03 06:09:28.043880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.043764    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1597ac30-419a-47f2-8802-15fa12ac07f9" containerName="revoke-keys"
Nov 03 06:09:28.054220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.054192    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"]
Nov 03 06:09:28.054375 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.054311    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.069619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.069588    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:28.113079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.112068    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:28.113079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.112192    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:28.113079 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.112315    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:28.165287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.159248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbx28\" (UniqueName: \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.165287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.159455    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.165287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.159708    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.261083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.261037    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbx28\" (UniqueName: \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.261251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.261183    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.261472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.261370    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.262075 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.262049    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.266404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.266382    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.323826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.316572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbx28\" (UniqueName: \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\") pod \"maas-api-revoke-sub-440f6461-sbprc\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.444947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.444911    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:28.742991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.742595    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:28.742991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.742651    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:28.865512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.865483    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"]
Nov 03 06:09:28.947816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.947610    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:28.947816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:28.947673    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:29.075158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:29.075120    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc" event={"ID":"bb112c3f-26a0-460f-b06c-608a1764e49b","Type":"ContainerStarted","Data":"862a5977050a8ab5f1737f11910be97132fd3be375276488731de750fd67572a"}
Nov 03 06:09:29.738759 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:29.738665    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:29.739043 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:29.739008    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:29.947688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:29.947639    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:29.947992 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:29.947943    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:30.143145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.143091    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc" event={"ID":"bb112c3f-26a0-460f-b06c-608a1764e49b","Type":"ContainerStarted","Data":"4bc5f5d8c479c0992a7ef6919c578ce5ba1dab7982550564456bf5f30bbf7cd9"}
Nov 03 06:09:30.204816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.204718    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc" podStartSLOduration=3.204694639 podStartE2EDuration="3.204694639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:30.182176025 +0000 UTC m=+858.593573104" watchObservedRunningTime="2026-11-03 06:09:30.204694639 +0000 UTC m=+858.616091719"
Nov 03 06:09:30.739274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.738769    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:30.739274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.738850    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:30.947385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.947343    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:30.947581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:30.947401    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:31.211975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.211935    4528 generic.go:356] "Generic (PLEG): container finished" podID="bb112c3f-26a0-460f-b06c-608a1764e49b" containerID="4bc5f5d8c479c0992a7ef6919c578ce5ba1dab7982550564456bf5f30bbf7cd9" exitCode=0
Nov 03 06:09:31.224635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.212733    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc" event={"ID":"bb112c3f-26a0-460f-b06c-608a1764e49b","Type":"ContainerDied","Data":"4bc5f5d8c479c0992a7ef6919c578ce5ba1dab7982550564456bf5f30bbf7cd9"}
Nov 03 06:09:31.739827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.739578    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused" start-of-body=
Nov 03 06:09:31.739827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.739645    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": dial tcp 10.132.0.122:15021: connect: connection refused"
Nov 03 06:09:31.947052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.947019    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused" start-of-body=
Nov 03 06:09:31.947254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:31.947080    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": dial tcp 10.132.0.123:15021: connect: connection refused"
Nov 03 06:09:32.524253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.524222    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:32.597596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.597556    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\") pod \"bb112c3f-26a0-460f-b06c-608a1764e49b\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") "
Nov 03 06:09:32.597780 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.597650    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\") pod \"bb112c3f-26a0-460f-b06c-608a1764e49b\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") "
Nov 03 06:09:32.597917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.597897    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\" (UniqueName: \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\") pod \"bb112c3f-26a0-460f-b06c-608a1764e49b\" (UID: \"bb112c3f-26a0-460f-b06c-608a1764e49b\") "
Nov 03 06:09:32.600256 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.600221    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca" pod "bb112c3f-26a0-460f-b06c-608a1764e49b" (UID: "bb112c3f-26a0-460f-b06c-608a1764e49b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:32.603181 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.603149    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca" pod "bb112c3f-26a0-460f-b06c-608a1764e49b" (UID: "bb112c3f-26a0-460f-b06c-608a1764e49b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:32.665264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.665223    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28" pod "bb112c3f-26a0-460f-b06c-608a1764e49b" (UID: "bb112c3f-26a0-460f-b06c-608a1764e49b"). InnerVolumeSpecName "kube-api-access-jbx28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:32.700936 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.700902    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:32.701184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.701165    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb112c3f-26a0-460f-b06c-608a1764e49b-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:32.701307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.701293    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbx28\" (UniqueName: \"kubernetes.io/projected/bb112c3f-26a0-460f-b06c-608a1764e49b-kube-api-access-jbx28\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:32.823492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.822272    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:32.823492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.823179    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:32.824603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.824578    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:09:32.952991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.952954    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:32.953484 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.953455    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:32.954723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:32.954682    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:09:33.241707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:33.241619    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"
Nov 03 06:09:33.241707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:33.241649    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc" event={"ID":"bb112c3f-26a0-460f-b06c-608a1764e49b","Type":"ContainerDied","Data":"862a5977050a8ab5f1737f11910be97132fd3be375276488731de750fd67572a"}
Nov 03 06:09:33.241707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:33.241706    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="862a5977050a8ab5f1737f11910be97132fd3be375276488731de750fd67572a"
Nov 03 06:09:38.259414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.259375    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"]
Nov 03 06:09:38.260143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.260127    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bb112c3f-26a0-460f-b06c-608a1764e49b" containerName="revoke-keys"
Nov 03 06:09:38.260207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.260144    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb112c3f-26a0-460f-b06c-608a1764e49b" containerName="revoke-keys"
Nov 03 06:09:38.260344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.260332    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb112c3f-26a0-460f-b06c-608a1764e49b" containerName="revoke-keys"
Nov 03 06:09:38.264572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.264550    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.284904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.284873    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"]
Nov 03 06:09:38.288452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.286933    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:38.288452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.287563    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:38.288452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.287599    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:38.288452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.287634    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:38.392688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.392651    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.392925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.392882    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jdbr\" (UniqueName: \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.393001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.392932    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.495040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.495003    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jdbr\" (UniqueName: \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.495246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.495076    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.495319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.495254    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.495926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.495902    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.497612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.497583    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.518364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.518289    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jdbr\" (UniqueName: \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\") pod \"maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.587526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.587483    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:38.609629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.609594    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"]
Nov 03 06:09:38.616350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.616309    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.635115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.635077    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"]
Nov 03 06:09:38.699819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.699594    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.699819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.699632    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5tr7\" (UniqueName: \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.700044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.699819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.802472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.802432    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.802660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.802486    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5tr7\" (UniqueName: \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.802660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.802602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.803335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.803306    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.805052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.805019    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.825683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.825655    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5tr7\" (UniqueName: \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\") pod \"maas-api-revoke-sub-b088b577-zdzdn\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:38.914778 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:38.914743    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1c47c9_52ff_4861_9907_91a7596d2c90.slice/crio-90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a WatchSource:0}: Error finding container 90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a: Status 404 returned error can't find the container with id 90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a
Nov 03 06:09:38.915474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.915448    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"]
Nov 03 06:09:38.953147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:38.953116    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:39.268163 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:39.268130    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dbd1c94_a96b_4420_bb10_5bede5215c7d.slice/crio-547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41 WatchSource:0}: Error finding container 547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41: Status 404 returned error can't find the container with id 547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41
Nov 03 06:09:39.272471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:39.272443    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"]
Nov 03 06:09:39.312028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:39.311994    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn" event={"ID":"3dbd1c94-a96b-4420-bb10-5bede5215c7d","Type":"ContainerStarted","Data":"547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41"}
Nov 03 06:09:39.313723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:39.313696    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4" event={"ID":"4d1c47c9-52ff-4861-9907-91a7596d2c90","Type":"ContainerStarted","Data":"90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a"}
Nov 03 06:09:40.326926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:40.326885    4528 generic.go:356] "Generic (PLEG): container finished" podID="3dbd1c94-a96b-4420-bb10-5bede5215c7d" containerID="687289b84ff7b7520365add6df41535f69f0fe124e86582d5c8d715904ba122e" exitCode=0
Nov 03 06:09:40.327340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:40.326965    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn" event={"ID":"3dbd1c94-a96b-4420-bb10-5bede5215c7d","Type":"ContainerDied","Data":"687289b84ff7b7520365add6df41535f69f0fe124e86582d5c8d715904ba122e"}
Nov 03 06:09:40.329214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:40.329187    4528 generic.go:356] "Generic (PLEG): container finished" podID="4d1c47c9-52ff-4861-9907-91a7596d2c90" containerID="0329d448fc587a67837660b15ad54e47f12a55895e38f1d6b663fd3b997932b6" exitCode=0
Nov 03 06:09:40.329334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:40.329265    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4" event={"ID":"4d1c47c9-52ff-4861-9907-91a7596d2c90","Type":"ContainerDied","Data":"0329d448fc587a67837660b15ad54e47f12a55895e38f1d6b663fd3b997932b6"}
Nov 03 06:09:42.452822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.449668    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:42.612850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.612817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\") pod \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") "
Nov 03 06:09:42.613128 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.613110    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\" (UniqueName: \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\") pod \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") "
Nov 03 06:09:42.613443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.613417    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\") pod \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\" (UID: \"3dbd1c94-a96b-4420-bb10-5bede5215c7d\") "
Nov 03 06:09:42.667903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.667782    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca" pod "3dbd1c94-a96b-4420-bb10-5bede5215c7d" (UID: "3dbd1c94-a96b-4420-bb10-5bede5215c7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:42.668528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.668487    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca" pod "3dbd1c94-a96b-4420-bb10-5bede5215c7d" (UID: "3dbd1c94-a96b-4420-bb10-5bede5215c7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:42.715002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.707449    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7" pod "3dbd1c94-a96b-4420-bb10-5bede5215c7d" (UID: "3dbd1c94-a96b-4420-bb10-5bede5215c7d"). InnerVolumeSpecName "kube-api-access-x5tr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:42.785817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.782215    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"
Nov 03 06:09:42.801816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.795343    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:42.801816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.795382    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dbd1c94-a96b-4420-bb10-5bede5215c7d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:42.801816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.795397    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5tr7\" (UniqueName: \"kubernetes.io/projected/3dbd1c94-a96b-4420-bb10-5bede5215c7d-kube-api-access-x5tr7\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:42.877864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.877746    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn" event={"ID":"3dbd1c94-a96b-4420-bb10-5bede5215c7d","Type":"ContainerDied","Data":"547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41"}
Nov 03 06:09:42.885432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.885312    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="547b14c427dd745cfcdff7686d1dd7e91e4794268584172801641059b74a9c41"
Nov 03 06:09:42.898038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:42.898004    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:43.041820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.028457    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\" (UniqueName: \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\") pod \"4d1c47c9-52ff-4861-9907-91a7596d2c90\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") "
Nov 03 06:09:43.041820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.028511    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\") pod \"4d1c47c9-52ff-4861-9907-91a7596d2c90\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") "
Nov 03 06:09:43.041820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.028759    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\") pod \"4d1c47c9-52ff-4861-9907-91a7596d2c90\" (UID: \"4d1c47c9-52ff-4861-9907-91a7596d2c90\") "
Nov 03 06:09:43.069196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.060374    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr" pod "4d1c47c9-52ff-4861-9907-91a7596d2c90" (UID: "4d1c47c9-52ff-4861-9907-91a7596d2c90"). InnerVolumeSpecName "kube-api-access-5jdbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:43.069196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.068896    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:43.092684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.092646    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca" pod "4d1c47c9-52ff-4861-9907-91a7596d2c90" (UID: "4d1c47c9-52ff-4861-9907-91a7596d2c90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:43.113554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.107879    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:09:43.157816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.156989    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jdbr\" (UniqueName: \"kubernetes.io/projected/4d1c47c9-52ff-4861-9907-91a7596d2c90-kube-api-access-5jdbr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:43.157816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.157483    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:43.344819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.344459    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca" pod "4d1c47c9-52ff-4861-9907-91a7596d2c90" (UID: "4d1c47c9-52ff-4861-9907-91a7596d2c90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:43.371817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.371615    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d1c47c9-52ff-4861-9907-91a7596d2c90-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:43.464826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.464295    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:09:43.480822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.464652    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:43.507056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.506926    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-54b885cf97-5hk8f" podUID="680c9d72-4f8d-49c0-95cd-d25aab3710ef" containerName="manager" probeResult="failure" output="Get \"http://10.132.0.33:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:43.713874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.713379    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:43.933821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:43.928896    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.030907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.007881    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.174820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.078158    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.174820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.094173    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.251856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.212385    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"
Nov 03 06:09:44.304885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.300114    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.304885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.300554    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:09:44.518828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.497984    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.583139 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.510442    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:09:44.583139 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.516921    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": context deadline exceeded"
Nov 03 06:09:44.667176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.666924    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:09:44.780013    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.02s"
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.780056    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4" event={"ID":"4d1c47c9-52ff-4861-9907-91a7596d2c90","Type":"ContainerDied","Data":"90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a"}
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.780086    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90cbf27fb7ce3ee495b9fe30cc9fbd87bbadee771e60c4a5b0abdba22116176a"
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.780264    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.780305    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:44.784756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.780341    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:44.786529 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:09:44.786496    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:09:44.977867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.959815    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:09:44.977867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:44.967663    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:45.032817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.025439    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.103851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.077921    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:09:45.140885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.129711    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.178990 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.157498    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:45.240279 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.240181    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded"
Nov 03 06:09:45.332819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.310484    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.335132 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.318155    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:09:45.371401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.348367    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:45.371401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.348443    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:45.450858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.442932    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:45.450858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.442999    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.524182 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.524074    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.589276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.589224    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.590542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.590490    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.678807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.678741    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:45.682818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.679350    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:45.984130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:45.973297    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:46.081816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.081163    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:09:46.360825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.360253    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:09:46.420822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.403751    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:46.420822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.404356    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": context deadline exceeded"
Nov 03 06:09:46.420822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.404834    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:09:46.579828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.579240    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:46.618880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.594215    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:46.685846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.681041    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 06:09:46.902877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:46.864390    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.088822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.056898    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.163756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.163651    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.466040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.465938    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.534263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.534203    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.614021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.613966    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded"
Nov 03 06:09:47.629272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.629226    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:47.631594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.631559    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:09:47.650345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.650305    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:09:47.684895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.684836    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:47.816678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:47.793562    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.149715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.149594    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.150013 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:09:48.029001    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.156s"
Nov 03 06:09:48.161864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.093149    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.162144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.014666    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.224188 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.224136    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:09:48.241287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.241239    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.241563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.241531    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.241762 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.241744    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.242051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.242029    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.375255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.375207    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.375578 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.375531    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.533874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.524646    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.580873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.579320    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.660921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.658908    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.660921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.658983    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.105:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.660921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.659071    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:09:48.660921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.660583    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.660921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.660904    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:09:48.661840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.661786    4528 patch_prober.go:29] interesting pod/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:48.662008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.661989    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:48.696806 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:48.694612    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.003722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.003340    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:09:49.003722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.003405    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.193528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.192943    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.256577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.245537    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:09:49.256577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.245567    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.362945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.362890    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded"
Nov 03 06:09:49.547898 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.547740    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.610141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.610089    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.722819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.664044    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.722819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.676036    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.722819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.677261    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:49.783636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.748876    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": context deadline exceeded"
Nov 03 06:09:49.783636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:49.766498    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.209924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.209877    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.330676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.330631    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:09:50.363586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.341112    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.548872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.548420    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.554450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.554395    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"]
Nov 03 06:09:50.588883 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.588840    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3dbd1c94-a96b-4420-bb10-5bede5215c7d" containerName="revoke-keys"
Nov 03 06:09:50.589249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.589214    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dbd1c94-a96b-4420-bb10-5bede5215c7d" containerName="revoke-keys"
Nov 03 06:09:50.589547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.589532    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4d1c47c9-52ff-4861-9907-91a7596d2c90" containerName="revoke-keys"
Nov 03 06:09:50.589679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.589664    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d1c47c9-52ff-4861-9907-91a7596d2c90" containerName="revoke-keys"
Nov 03 06:09:50.599041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.599012    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dbd1c94-a96b-4420-bb10-5bede5215c7d" containerName="revoke-keys"
Nov 03 06:09:50.599266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.599251    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d1c47c9-52ff-4861-9907-91a7596d2c90" containerName="revoke-keys"
Nov 03 06:09:50.605910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.605879    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.648828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.634343    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhjsk\" (UniqueName: \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.656828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.649877    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.677811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.657725    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.681094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.646986    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:09:50.681399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.681379    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:09:50.687420 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.647419    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:09:50.688724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.679207    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.689546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.680128    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:09:50.689837 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.680746    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:09:50.690359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.680850    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:09:50.703361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.702758    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:09:50.708776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.708213    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:09:50.708776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.708307    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:09:50.711416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.711393    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:09:50.730288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.730258    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"]
Nov 03 06:09:50.738918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.738204    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:09:50.752759 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.752720    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:09:50.761823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.756484    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:09:50.761823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.757929    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:09:50.763008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.762986    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:09:50.782283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.782240    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.782456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.782341    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.782870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.782844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhjsk\" (UniqueName: \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.784230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.784203    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.787469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.787444    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:50.807254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:50.807218    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhjsk\" (UniqueName: \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\") pod \"maas-api-revoke-sub-932625f0-2l2nc\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:51.060268 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:51.060182    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:51.394786 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:51.394752    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod038fb417_ce3c_4c44_85ea_cdd567b8d94a.slice/crio-0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f WatchSource:0}: Error finding container 0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f: Status 404 returned error can't find the container with id 0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f
Nov 03 06:09:51.397173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:51.396986    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"]
Nov 03 06:09:52.298506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.298469    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc" event={"ID":"038fb417-ce3c-4c44-85ea-cdd567b8d94a","Type":"ContainerStarted","Data":"60f9eb3a53b5542e850686e164f4e14ae52c5361bc63a4c78ca0cd2a6b44192e"}
Nov 03 06:09:52.299064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.298525    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc" event={"ID":"038fb417-ce3c-4c44-85ea-cdd567b8d94a","Type":"ContainerStarted","Data":"0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f"}
Nov 03 06:09:52.345075 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.345014    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc" podStartSLOduration=4.34500079 podStartE2EDuration="4.34500079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:52.337928177 +0000 UTC m=+880.749325254" watchObservedRunningTime="2026-11-03 06:09:52.34500079 +0000 UTC m=+880.756397867"
Nov 03 06:09:52.441464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.441425    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"]
Nov 03 06:09:52.446238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.446220    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.466869 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.466783    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"]
Nov 03 06:09:52.519346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.519308    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.519538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.519383    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggt4g\" (UniqueName: \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.519538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.519519    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.621509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.621470    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.621751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.621543    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggt4g\" (UniqueName: \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.621751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.621637    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.622517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.622493    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.624115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.624085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.646144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.646121    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggt4g\" (UniqueName: \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\") pod \"maas-api-revoke-sub-4d635ee9-6bfzr\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:52.761553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:52.761513    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:53.110888 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:53.103280    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bb1afc_8271_488d_97b4_8360da80568d.slice/crio-b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6 WatchSource:0}: Error finding container b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6: Status 404 returned error can't find the container with id b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6
Nov 03 06:09:53.110888 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:53.104572    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"]
Nov 03 06:09:53.329212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:53.329168    4528 generic.go:356] "Generic (PLEG): container finished" podID="038fb417-ce3c-4c44-85ea-cdd567b8d94a" containerID="60f9eb3a53b5542e850686e164f4e14ae52c5361bc63a4c78ca0cd2a6b44192e" exitCode=0
Nov 03 06:09:53.329721 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:53.329547    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc" event={"ID":"038fb417-ce3c-4c44-85ea-cdd567b8d94a","Type":"ContainerDied","Data":"60f9eb3a53b5542e850686e164f4e14ae52c5361bc63a4c78ca0cd2a6b44192e"}
Nov 03 06:09:53.332312 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:53.332278    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr" event={"ID":"03bb1afc-8271-488d-97b4-8360da80568d","Type":"ContainerStarted","Data":"b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6"}
Nov 03 06:09:54.346500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.346466    4528 generic.go:356] "Generic (PLEG): container finished" podID="03bb1afc-8271-488d-97b4-8360da80568d" containerID="7a8089ec9d28335b73c4b5d1a32ad28b3014c712923b7c9f2fb87abcd78fd414" exitCode=0
Nov 03 06:09:54.347062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.346565    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr" event={"ID":"03bb1afc-8271-488d-97b4-8360da80568d","Type":"ContainerDied","Data":"7a8089ec9d28335b73c4b5d1a32ad28b3014c712923b7c9f2fb87abcd78fd414"}
Nov 03 06:09:54.389523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.389490    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"]
Nov 03 06:09:54.395216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.394587    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.413914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.413878    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"]
Nov 03 06:09:54.556069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.556036    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.556258 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.556227    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.556327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.556306    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfqj\" (UniqueName: \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.563544 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.563517    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:54.635228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.635131    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"]
Nov 03 06:09:54.635987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.635960    4528 cpu_manager.go:414] "containerMap: removing container" podUID="038fb417-ce3c-4c44-85ea-cdd567b8d94a" containerName="revoke-keys"
Nov 03 06:09:54.635987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.635980    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="038fb417-ce3c-4c44-85ea-cdd567b8d94a" containerName="revoke-keys"
Nov 03 06:09:54.636182 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.636136    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="038fb417-ce3c-4c44-85ea-cdd567b8d94a" containerName="revoke-keys"
Nov 03 06:09:54.640746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.640718    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.658522    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\") pod \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") "
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.658888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\" (UniqueName: \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\") pod \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") "
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.659015    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\") pod \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\" (UID: \"038fb417-ce3c-4c44-85ea-cdd567b8d94a\") "
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.659457    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.659594    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.660041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.659658    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfqj\" (UniqueName: \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.660413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.660259    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca" pod "038fb417-ce3c-4c44-85ea-cdd567b8d94a" (UID: "038fb417-ce3c-4c44-85ea-cdd567b8d94a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:54.662394 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.662369    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk" pod "038fb417-ce3c-4c44-85ea-cdd567b8d94a" (UID: "038fb417-ce3c-4c44-85ea-cdd567b8d94a"). InnerVolumeSpecName "kube-api-access-nhjsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:54.664338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.664307    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.664689 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.664666    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca" pod "038fb417-ce3c-4c44-85ea-cdd567b8d94a" (UID: "038fb417-ce3c-4c44-85ea-cdd567b8d94a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:54.664866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.664845    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"]
Nov 03 06:09:54.667851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.667828    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.680845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.673220    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"]
Nov 03 06:09:54.698971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.698934    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfqj\" (UniqueName: \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\") pod \"maas-api-revoke-sub-2e007df9-c5mmk\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.705148 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.705113    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.715672 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.715641    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"]
Nov 03 06:09:54.720544 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.720512    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:54.761074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761043    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hmbt\" (UniqueName: \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.761191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761113    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.761359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761335    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.761491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761476    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhjsk\" (UniqueName: \"kubernetes.io/projected/038fb417-ce3c-4c44-85ea-cdd567b8d94a-kube-api-access-nhjsk\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:54.761556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761498    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:54.761556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.761516    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/038fb417-ce3c-4c44-85ea-cdd567b8d94a-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:54.863141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863094    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.863338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863280    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.863404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863362    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.863404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw5qc\" (UniqueName: \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.865638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863568    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hmbt\" (UniqueName: \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.865638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.863745    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.865638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.864046    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.867381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.867336    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.887666 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.887588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hmbt\" (UniqueName: \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\") pod \"maas-api-revoke-sub-93228110-cc6g6\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:54.965881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.965847    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.966087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.965889    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw5qc\" (UniqueName: \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.966227 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.966207    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.966653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.966625    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.968745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.968724    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:54.988695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:54.988671    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw5qc\" (UniqueName: \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\") pod \"maas-api-revoke-sub-24e53cd5-58m7h\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:55.033849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.033783    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:55.040301 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.040277    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:55.092543 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.088946    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"]
Nov 03 06:09:55.358610 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.358568    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk" event={"ID":"651f869b-7616-4abf-8e56-3ef001213926","Type":"ContainerStarted","Data":"a39b8e126c111b5655a30a389062d0dfbf2d6b10623dfdae742a7b5ce7947cd2"}
Nov 03 06:09:55.360130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.360107    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"
Nov 03 06:09:55.360130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.360116    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc" event={"ID":"038fb417-ce3c-4c44-85ea-cdd567b8d94a","Type":"ContainerDied","Data":"0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f"}
Nov 03 06:09:55.360271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.360170    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f868eb9870521b494c0716466692ba6ed0a28dea24aebc80c7f7541f26a336f"
Nov 03 06:09:55.399618 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:09:55.399570    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod766bd37a_3a4a_4b45_bc41_2ca774589377.slice/crio-bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4 WatchSource:0}: Error finding container bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4: Status 404 returned error can't find the container with id bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4
Nov 03 06:09:55.400432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.400403    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"]
Nov 03 06:09:55.465350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.465244    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"]
Nov 03 06:09:55.671954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.671926    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:55.758648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.758567    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing" containerID="cri-o://f8f4ce8fee0507364f03470ac7d89c5479aec39c024fc847d1d4827d9bfdcea8" gracePeriod=25
Nov 03 06:09:55.783149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.783122    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\") pod \"03bb1afc-8271-488d-97b4-8360da80568d\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") "
Nov 03 06:09:55.783323 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.783279    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\" (UniqueName: \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\") pod \"03bb1afc-8271-488d-97b4-8360da80568d\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") "
Nov 03 06:09:55.783388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.783324    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\") pod \"03bb1afc-8271-488d-97b4-8360da80568d\" (UID: \"03bb1afc-8271-488d-97b4-8360da80568d\") "
Nov 03 06:09:55.783710 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.783686    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca" pod "03bb1afc-8271-488d-97b4-8360da80568d" (UID: "03bb1afc-8271-488d-97b4-8360da80568d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:55.784015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.783997    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:55.785186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.785163    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca" pod "03bb1afc-8271-488d-97b4-8360da80568d" (UID: "03bb1afc-8271-488d-97b4-8360da80568d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:55.785424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.785404    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g" pod "03bb1afc-8271-488d-97b4-8360da80568d" (UID: "03bb1afc-8271-488d-97b4-8360da80568d"). InnerVolumeSpecName "kube-api-access-ggt4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:55.885440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.885403    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/03bb1afc-8271-488d-97b4-8360da80568d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:55.885440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:55.885435    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggt4g\" (UniqueName: \"kubernetes.io/projected/03bb1afc-8271-488d-97b4-8360da80568d-kube-api-access-ggt4g\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:56.371235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.371194    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6" event={"ID":"766bd37a-3a4a-4b45-bc41-2ca774589377","Type":"ContainerStarted","Data":"06b04436640f726c514106dff4678cabeec2770db73af11edbd129e7d87dc067"}
Nov 03 06:09:56.371680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.371264    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6" event={"ID":"766bd37a-3a4a-4b45-bc41-2ca774589377","Type":"ContainerStarted","Data":"bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4"}
Nov 03 06:09:56.373129 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.373099    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h" event={"ID":"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e","Type":"ContainerStarted","Data":"8c793e6e9c48cd49724e71024f9931252079ea8a2ce4c2891f0f760cd32c3ccc"}
Nov 03 06:09:56.373294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.373143    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h" event={"ID":"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e","Type":"ContainerStarted","Data":"3012103e58b39b5d57e0550087e1686df6a6eb2336501ef7fd42eee95a6a7b12"}
Nov 03 06:09:56.374713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.374692    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"
Nov 03 06:09:56.374852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.374730    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr" event={"ID":"03bb1afc-8271-488d-97b4-8360da80568d","Type":"ContainerDied","Data":"b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6"}
Nov 03 06:09:56.374852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.374774    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b223b45c5e38d7d4759dda823450724522eff8b216931293f5ef8d1b3914cbd6"
Nov 03 06:09:56.384187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.384150    4528 generic.go:356] "Generic (PLEG): container finished" podID="651f869b-7616-4abf-8e56-3ef001213926" containerID="47254c3d8f879634fe31882132347e50ad608b4700f2fa47f3bd897ebbc6c691" exitCode=0
Nov 03 06:09:56.384313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.384221    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk" event={"ID":"651f869b-7616-4abf-8e56-3ef001213926","Type":"ContainerDied","Data":"47254c3d8f879634fe31882132347e50ad608b4700f2fa47f3bd897ebbc6c691"}
Nov 03 06:09:56.419205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.419158    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6" podStartSLOduration=2.419144696 podStartE2EDuration="2.419144696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:56.412140944 +0000 UTC m=+884.823538023" watchObservedRunningTime="2026-11-03 06:09:56.419144696 +0000 UTC m=+884.830541773"
Nov 03 06:09:56.456116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:56.456064    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h" podStartSLOduration=2.456050761 podStartE2EDuration="2.456050761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:09:56.452416014 +0000 UTC m=+884.863813092" watchObservedRunningTime="2026-11-03 06:09:56.456050761 +0000 UTC m=+884.867447838"
Nov 03 06:09:57.397230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.397196    4528 generic.go:356] "Generic (PLEG): container finished" podID="766bd37a-3a4a-4b45-bc41-2ca774589377" containerID="06b04436640f726c514106dff4678cabeec2770db73af11edbd129e7d87dc067" exitCode=0
Nov 03 06:09:57.397712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.397280    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6" event={"ID":"766bd37a-3a4a-4b45-bc41-2ca774589377","Type":"ContainerDied","Data":"06b04436640f726c514106dff4678cabeec2770db73af11edbd129e7d87dc067"}
Nov 03 06:09:57.399045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.399021    4528 generic.go:356] "Generic (PLEG): container finished" podID="c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" containerID="8c793e6e9c48cd49724e71024f9931252079ea8a2ce4c2891f0f760cd32c3ccc" exitCode=0
Nov 03 06:09:57.399160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.399106    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h" event={"ID":"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e","Type":"ContainerDied","Data":"8c793e6e9c48cd49724e71024f9931252079ea8a2ce4c2891f0f760cd32c3ccc"}
Nov 03 06:09:57.585561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.585531    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:57.713409 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.713319    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\") pod \"651f869b-7616-4abf-8e56-3ef001213926\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") "
Nov 03 06:09:57.713585 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.713470    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\") pod \"651f869b-7616-4abf-8e56-3ef001213926\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") "
Nov 03 06:09:57.713716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.713695    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\" (UniqueName: \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\") pod \"651f869b-7616-4abf-8e56-3ef001213926\" (UID: \"651f869b-7616-4abf-8e56-3ef001213926\") "
Nov 03 06:09:57.714057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.714025    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca" pod "651f869b-7616-4abf-8e56-3ef001213926" (UID: "651f869b-7616-4abf-8e56-3ef001213926"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:57.714612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.714591    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f869b-7616-4abf-8e56-3ef001213926-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:57.715631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.715605    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca" pod "651f869b-7616-4abf-8e56-3ef001213926" (UID: "651f869b-7616-4abf-8e56-3ef001213926"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:57.715774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.715753    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj" pod "651f869b-7616-4abf-8e56-3ef001213926" (UID: "651f869b-7616-4abf-8e56-3ef001213926"). InnerVolumeSpecName "kube-api-access-pnfqj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:57.816359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.816317    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f869b-7616-4abf-8e56-3ef001213926-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:57.816359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:57.816359    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfqj\" (UniqueName: \"kubernetes.io/projected/651f869b-7616-4abf-8e56-3ef001213926-kube-api-access-pnfqj\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.414067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.414032    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"
Nov 03 06:09:58.414067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.414059    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk" event={"ID":"651f869b-7616-4abf-8e56-3ef001213926","Type":"ContainerDied","Data":"a39b8e126c111b5655a30a389062d0dfbf2d6b10623dfdae742a7b5ce7947cd2"}
Nov 03 06:09:58.414605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.414107    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a39b8e126c111b5655a30a389062d0dfbf2d6b10623dfdae742a7b5ce7947cd2"
Nov 03 06:09:58.634378 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.634349    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:09:58.639604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.639576    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.737497    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\" (UniqueName: \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\") pod \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.737599    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\") pod \"766bd37a-3a4a-4b45-bc41-2ca774589377\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.737756    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\") pod \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.737981    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\" (UniqueName: \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\") pod \"766bd37a-3a4a-4b45-bc41-2ca774589377\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.738104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\") pod \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\" (UID: \"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.738189    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\") pod \"766bd37a-3a4a-4b45-bc41-2ca774589377\" (UID: \"766bd37a-3a4a-4b45-bc41-2ca774589377\") "
Nov 03 06:09:58.741393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.740604    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca" pod "766bd37a-3a4a-4b45-bc41-2ca774589377" (UID: "766bd37a-3a4a-4b45-bc41-2ca774589377"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:58.743925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.743699    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca" pod "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" (UID: "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:09:58.751712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.750928    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca" pod "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" (UID: "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:58.752714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.752668    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc" pod "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" (UID: "c048cd5e-c213-41df-8e2d-d5ee8ff17e5e"). InnerVolumeSpecName "kube-api-access-tw5qc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:58.753005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.752981    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca" pod "766bd37a-3a4a-4b45-bc41-2ca774589377" (UID: "766bd37a-3a4a-4b45-bc41-2ca774589377"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:09:58.797814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.794295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt" pod "766bd37a-3a4a-4b45-bc41-2ca774589377" (UID: "766bd37a-3a4a-4b45-bc41-2ca774589377"). InnerVolumeSpecName "kube-api-access-5hmbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:09:58.842624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842588    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.842624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842619    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.842624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842628    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw5qc\" (UniqueName: \"kubernetes.io/projected/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-kube-api-access-tw5qc\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.843000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842637    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766bd37a-3a4a-4b45-bc41-2ca774589377-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.843000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842646    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:58.843000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:58.842658    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hmbt\" (UniqueName: \"kubernetes.io/projected/766bd37a-3a4a-4b45-bc41-2ca774589377-kube-api-access-5hmbt\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:09:59.430216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.430070    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6" event={"ID":"766bd37a-3a4a-4b45-bc41-2ca774589377","Type":"ContainerDied","Data":"bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4"}
Nov 03 06:09:59.430702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.430284    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf647cca15da86dbb28d2297d7b19d2b5c1efe2b371d3dd7b9448c50dc53eab4"
Nov 03 06:09:59.430702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.430393    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"
Nov 03 06:09:59.434264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.434223    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h" event={"ID":"c048cd5e-c213-41df-8e2d-d5ee8ff17e5e","Type":"ContainerDied","Data":"3012103e58b39b5d57e0550087e1686df6a6eb2336501ef7fd42eee95a6a7b12"}
Nov 03 06:09:59.434411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.434292    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3012103e58b39b5d57e0550087e1686df6a6eb2336501ef7fd42eee95a6a7b12"
Nov 03 06:09:59.434411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:09:59.434395    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"
Nov 03 06:10:02.419235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:02.419194    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:03.722505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.722470    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"]
Nov 03 06:10:03.723301 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723282    4528 cpu_manager.go:414] "containerMap: removing container" podUID="766bd37a-3a4a-4b45-bc41-2ca774589377" containerName="revoke-keys"
Nov 03 06:10:03.723355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723303    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766bd37a-3a4a-4b45-bc41-2ca774589377" containerName="revoke-keys"
Nov 03 06:10:03.723355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723322    4528 cpu_manager.go:414] "containerMap: removing container" podUID="03bb1afc-8271-488d-97b4-8360da80568d" containerName="revoke-keys"
Nov 03 06:10:03.723355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723328    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bb1afc-8271-488d-97b4-8360da80568d" containerName="revoke-keys"
Nov 03 06:10:03.723355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723345    4528 cpu_manager.go:414] "containerMap: removing container" podUID="651f869b-7616-4abf-8e56-3ef001213926" containerName="revoke-keys"
Nov 03 06:10:03.723355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723351    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651f869b-7616-4abf-8e56-3ef001213926" containerName="revoke-keys"
Nov 03 06:10:03.723525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723367    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" containerName="revoke-keys"
Nov 03 06:10:03.723525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723375    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" containerName="revoke-keys"
Nov 03 06:10:03.723525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723493    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" containerName="revoke-keys"
Nov 03 06:10:03.723525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723508    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="766bd37a-3a4a-4b45-bc41-2ca774589377" containerName="revoke-keys"
Nov 03 06:10:03.723525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723517    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bb1afc-8271-488d-97b4-8360da80568d" containerName="revoke-keys"
Nov 03 06:10:03.723679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.723528    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="651f869b-7616-4abf-8e56-3ef001213926" containerName="revoke-keys"
Nov 03 06:10:03.728896 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.728875    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.748240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.748207    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:10:03.748387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.748236    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:10:03.748387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.748323    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:10:03.748492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.748413    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:10:03.760171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.760137    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"]
Nov 03 06:10:03.825123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.825090    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.825297 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.825273    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nph8m\" (UniqueName: \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.825349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.825309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.929064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.927129    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.929064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.927308    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nph8m\" (UniqueName: \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.929064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.927356    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.929064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.928599    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.931515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.931489    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:03.952949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:03.952910    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nph8m\" (UniqueName: \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\") pod \"maas-api-revoke-sub-454693e6-czm7s\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:04.051985 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.051957    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:04.112996 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.111632    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" containerID="cri-o://050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864" gracePeriod=39
Nov 03 06:10:04.375984 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:04.375858    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4b7e6b_4199_4928_a724_6064c2e1aa0a.slice/crio-a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed WatchSource:0}: Error finding container a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed: Status 404 returned error can't find the container with id a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed
Nov 03 06:10:04.384373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.382067    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"]
Nov 03 06:10:04.405211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.405182    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:10:04.435858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.435824    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436034    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436091    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:04.436271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436103    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436194    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436263    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\" (UniqueName: \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436285    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:04.436687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\") pod \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\" (UID: \"1c1d658e-c4c7-425e-9b83-b3431fbcff06\") "
Nov 03 06:10:04.436687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.436498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:04.437107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.437087    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.437194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.437111    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.437194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.437127    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.438531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.438497    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:04.438613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.438584    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:04.438770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.438750    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9" pod "1c1d658e-c4c7-425e-9b83-b3431fbcff06" (UID: "1c1d658e-c4c7-425e-9b83-b3431fbcff06"). InnerVolumeSpecName "kube-api-access-kpsf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:04.511308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.511259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s" event={"ID":"5d4b7e6b-4199-4928-a724-6064c2e1aa0a","Type":"ContainerStarted","Data":"a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed"}
Nov 03 06:10:04.515366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.515330    4528 generic.go:356] "Generic (PLEG): container finished" podID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerID="050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864" exitCode=0
Nov 03 06:10:04.515552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.515382    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" event={"ID":"1c1d658e-c4c7-425e-9b83-b3431fbcff06","Type":"ContainerDied","Data":"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"}
Nov 03 06:10:04.515552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.515429    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" event={"ID":"1c1d658e-c4c7-425e-9b83-b3431fbcff06","Type":"ContainerDied","Data":"5217a017c725324eb7a4994e0a6afa20b0e0f101d725aad84dd3e908718c7850"}
Nov 03 06:10:04.515552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.515451    4528 scope.go:122] "RemoveContainer" containerID="050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"
Nov 03 06:10:04.516707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.516163    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"
Nov 03 06:10:04.534108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.534080    4528 scope.go:122] "RemoveContainer" containerID="050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"
Nov 03 06:10:04.534550 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:04.534519    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864\": container with ID starting with 050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864 not found: ID does not exist" containerID="050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"
Nov 03 06:10:04.534646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.534563    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864"} err="failed to get container status \"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864\": rpc error: code = NotFound desc = could not find container \"050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864\": container with ID starting with 050ae4b30611810f8d1e32d32c1cc9581ffe860cc89f9c88ae0c06518c099864 not found: ID does not exist"
Nov 03 06:10:04.538965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.538940    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c1d658e-c4c7-425e-9b83-b3431fbcff06-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.538965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.538963    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpsf9\" (UniqueName: \"kubernetes.io/projected/1c1d658e-c4c7-425e-9b83-b3431fbcff06-kube-api-access-kpsf9\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.539145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.538975    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c1d658e-c4c7-425e-9b83-b3431fbcff06-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:04.587556 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.587490    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:10:04.595460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.595427    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422"]
Nov 03 06:10:04.987050 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.987015    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"]
Nov 03 06:10:04.987770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.987753    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main"
Nov 03 06:10:04.987770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.987769    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main"
Nov 03 06:10:04.987975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.987901    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main"
Nov 03 06:10:04.992039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:04.992018    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.010098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.010065    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"]
Nov 03 06:10:05.049497 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.049451    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.049715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.049617    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.049879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.049847    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mtzs\" (UniqueName: \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.151642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.151591    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.151910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.151672    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.151910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.151747    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mtzs\" (UniqueName: \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.152537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.152508    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.154086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.154061    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.174831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.174783    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mtzs\" (UniqueName: \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\") pod \"maas-api-revoke-sub-6b2757ba-w6fhk\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.324032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.323997    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:05.329703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.329665    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-b7e447/premium-w3-b7e447-kserve-869b76c5d-qs422" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" containerName="main" probeResult="failure" output="Get \"https://10.132.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:05.529994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.529960    4528 generic.go:356] "Generic (PLEG): container finished" podID="5d4b7e6b-4199-4928-a724-6064c2e1aa0a" containerID="afc2f158f54d08e22d179f46b25e056bad76109efcb72f98e794849296521b01" exitCode=0
Nov 03 06:10:05.530174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.530041    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s" event={"ID":"5d4b7e6b-4199-4928-a724-6064c2e1aa0a","Type":"ContainerDied","Data":"afc2f158f54d08e22d179f46b25e056bad76109efcb72f98e794849296521b01"}
Nov 03 06:10:05.635283 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:05.635239    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e08e38_2bfd_4099_a13c_9d85807c9c2b.slice/crio-410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5 WatchSource:0}: Error finding container 410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5: Status 404 returned error can't find the container with id 410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5
Nov 03 06:10:05.636879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.636852    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"]
Nov 03 06:10:05.764749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:05.764711    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c1d658e-c4c7-425e-9b83-b3431fbcff06" path="/var/lib/kubelet/pods/1c1d658e-c4c7-425e-9b83-b3431fbcff06/volumes"
Nov 03 06:10:06.546483 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.546442    4528 generic.go:356] "Generic (PLEG): container finished" podID="c0e08e38-2bfd-4099-a13c-9d85807c9c2b" containerID="d7f80d10eb411b3bf157ca703cfe73786d323922c5df6872ae69ef705d93a532" exitCode=0
Nov 03 06:10:06.546941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.546524    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk" event={"ID":"c0e08e38-2bfd-4099-a13c-9d85807c9c2b","Type":"ContainerDied","Data":"d7f80d10eb411b3bf157ca703cfe73786d323922c5df6872ae69ef705d93a532"}
Nov 03 06:10:06.546941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.546586    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk" event={"ID":"c0e08e38-2bfd-4099-a13c-9d85807c9c2b","Type":"ContainerStarted","Data":"410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5"}
Nov 03 06:10:06.731733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.731703    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:06.781535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.781497    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\" (UniqueName: \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\") pod \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") "
Nov 03 06:10:06.781726 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.781625    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\") pod \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") "
Nov 03 06:10:06.781726 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.781674    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\") pod \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\" (UID: \"5d4b7e6b-4199-4928-a724-6064c2e1aa0a\") "
Nov 03 06:10:06.783423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.783392    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca" pod "5d4b7e6b-4199-4928-a724-6064c2e1aa0a" (UID: "5d4b7e6b-4199-4928-a724-6064c2e1aa0a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:06.785462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.785417    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca" pod "5d4b7e6b-4199-4928-a724-6064c2e1aa0a" (UID: "5d4b7e6b-4199-4928-a724-6064c2e1aa0a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:06.785815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.785764    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m" pod "5d4b7e6b-4199-4928-a724-6064c2e1aa0a" (UID: "5d4b7e6b-4199-4928-a724-6064c2e1aa0a"). InnerVolumeSpecName "kube-api-access-nph8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:06.885004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.884970    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nph8m\" (UniqueName: \"kubernetes.io/projected/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-kube-api-access-nph8m\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:06.885004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.885005    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:06.885248 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:06.885021    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d4b7e6b-4199-4928-a724-6064c2e1aa0a-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:07.124952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.124912    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"]
Nov 03 06:10:07.126657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.126603    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5d4b7e6b-4199-4928-a724-6064c2e1aa0a" containerName="revoke-keys"
Nov 03 06:10:07.126657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.126634    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d4b7e6b-4199-4928-a724-6064c2e1aa0a" containerName="revoke-keys"
Nov 03 06:10:07.127111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.127092    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d4b7e6b-4199-4928-a724-6064c2e1aa0a" containerName="revoke-keys"
Nov 03 06:10:07.135586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.135557    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.136727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.136701    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"]
Nov 03 06:10:07.189976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.189942    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.190152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.190049    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.190152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.190103    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5hww\" (UniqueName: \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.291949 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.291912    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.292151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.291976    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.292151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.292028    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5hww\" (UniqueName: \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.292748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.292719    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.294463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.294438    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.315342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.315306    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5hww\" (UniqueName: \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\") pod \"maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.453012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.452921    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:07.561962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.561923    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"
Nov 03 06:10:07.562318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.561957    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s" event={"ID":"5d4b7e6b-4199-4928-a724-6064c2e1aa0a","Type":"ContainerDied","Data":"a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed"}
Nov 03 06:10:07.562318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.562013    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31949602ba4e8ae866cb3727e6a417791027a9e34904c5b3f786c72cb57e3ed"
Nov 03 06:10:07.743491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.743467    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:07.765475 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:07.765448    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd7f922_a1f9_4bc8_995b_fbd2991589d9.slice/crio-38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3 WatchSource:0}: Error finding container 38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3: Status 404 returned error can't find the container with id 38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3
Nov 03 06:10:07.768846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.768703    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"]
Nov 03 06:10:07.814401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.814368    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\") pod \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") "
Nov 03 06:10:07.814591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.814572    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\" (UniqueName: \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\") pod \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") "
Nov 03 06:10:07.814710 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.814689    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\") pod \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\" (UID: \"c0e08e38-2bfd-4099-a13c-9d85807c9c2b\") "
Nov 03 06:10:07.815059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.815033    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca" pod "c0e08e38-2bfd-4099-a13c-9d85807c9c2b" (UID: "c0e08e38-2bfd-4099-a13c-9d85807c9c2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:07.815879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.815845    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:07.837852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.836579    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca" pod "c0e08e38-2bfd-4099-a13c-9d85807c9c2b" (UID: "c0e08e38-2bfd-4099-a13c-9d85807c9c2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:07.846437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.846386    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs" pod "c0e08e38-2bfd-4099-a13c-9d85807c9c2b" (UID: "c0e08e38-2bfd-4099-a13c-9d85807c9c2b"). InnerVolumeSpecName "kube-api-access-8mtzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:07.918094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.917979    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mtzs\" (UniqueName: \"kubernetes.io/projected/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-kube-api-access-8mtzs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:07.918094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:07.918015    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0e08e38-2bfd-4099-a13c-9d85807c9c2b-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:08.573067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.573026    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r" event={"ID":"fcd7f922-a1f9-4bc8-995b-fbd2991589d9","Type":"ContainerStarted","Data":"8dcb5a08c64d2a9e5a8db3070893920f96c7f2d6cf95efafc817d6bd32319ba9"}
Nov 03 06:10:08.573479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.573093    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r" event={"ID":"fcd7f922-a1f9-4bc8-995b-fbd2991589d9","Type":"ContainerStarted","Data":"38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3"}
Nov 03 06:10:08.574558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.574533    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk" event={"ID":"c0e08e38-2bfd-4099-a13c-9d85807c9c2b","Type":"ContainerDied","Data":"410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5"}
Nov 03 06:10:08.574699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.574572    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"
Nov 03 06:10:08.574699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.574574    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="410e3dc0cc11e131d2ad50f0ec96145df81907df66d918c2729e11597e76def5"
Nov 03 06:10:08.621626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:08.621552    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r" podStartSLOduration=1.621530406 podStartE2EDuration="1.621530406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:08.611640406 +0000 UTC m=+897.023037487" watchObservedRunningTime="2026-11-03 06:10:08.621530406 +0000 UTC m=+897.032927486"
Nov 03 06:10:09.586615 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.586580    4528 generic.go:356] "Generic (PLEG): container finished" podID="fcd7f922-a1f9-4bc8-995b-fbd2991589d9" containerID="8dcb5a08c64d2a9e5a8db3070893920f96c7f2d6cf95efafc817d6bd32319ba9" exitCode=0
Nov 03 06:10:09.587093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.586668    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r" event={"ID":"fcd7f922-a1f9-4bc8-995b-fbd2991589d9","Type":"ContainerDied","Data":"8dcb5a08c64d2a9e5a8db3070893920f96c7f2d6cf95efafc817d6bd32319ba9"}
Nov 03 06:10:09.754509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.754472    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"]
Nov 03 06:10:09.755401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.755291    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c0e08e38-2bfd-4099-a13c-9d85807c9c2b" containerName="revoke-keys"
Nov 03 06:10:09.755401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.755315    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0e08e38-2bfd-4099-a13c-9d85807c9c2b" containerName="revoke-keys"
Nov 03 06:10:09.755650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.755604    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0e08e38-2bfd-4099-a13c-9d85807c9c2b" containerName="revoke-keys"
Nov 03 06:10:09.761011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.760741    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.766840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.766787    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"]
Nov 03 06:10:09.850518 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.850427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.850653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.850607    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.850749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.850735    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsvv2\" (UniqueName: \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.952151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.952108    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsvv2\" (UniqueName: \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.952345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.952236    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.952399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.952353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.953116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.953088    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.954723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.954699    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:09.975125 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:09.975096    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsvv2\" (UniqueName: \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\") pod \"maas-api-revoke-sub-afbdb8f0-kxsp7\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:10.080333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.080292    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:10.397344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.397304    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"]
Nov 03 06:10:10.398539 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:10.398502    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c6dd176_5325_4ac3_b302_8c2cc615e7c6.slice/crio-8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da WatchSource:0}: Error finding container 8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da: Status 404 returned error can't find the container with id 8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da
Nov 03 06:10:10.601078 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.601038    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7" event={"ID":"9c6dd176-5325-4ac3-b302-8c2cc615e7c6","Type":"ContainerStarted","Data":"8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da"}
Nov 03 06:10:10.793426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.793398    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:10.967287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.967254    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\" (UniqueName: \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\") pod \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") "
Nov 03 06:10:10.967454 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.967404    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\") pod \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") "
Nov 03 06:10:10.967806 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.967749    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca" pod "fcd7f922-a1f9-4bc8-995b-fbd2991589d9" (UID: "fcd7f922-a1f9-4bc8-995b-fbd2991589d9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:10.967806 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.967757    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\") pod \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\" (UID: \"fcd7f922-a1f9-4bc8-995b-fbd2991589d9\") "
Nov 03 06:10:10.968538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.968512    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:10.969778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.969754    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww" pod "fcd7f922-a1f9-4bc8-995b-fbd2991589d9" (UID: "fcd7f922-a1f9-4bc8-995b-fbd2991589d9"). InnerVolumeSpecName "kube-api-access-x5hww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:10.970355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:10.969859    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca" pod "fcd7f922-a1f9-4bc8-995b-fbd2991589d9" (UID: "fcd7f922-a1f9-4bc8-995b-fbd2991589d9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:11.070157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.070122    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:11.070157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.070153    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5hww\" (UniqueName: \"kubernetes.io/projected/fcd7f922-a1f9-4bc8-995b-fbd2991589d9-kube-api-access-x5hww\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:11.614094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.614060    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"
Nov 03 06:10:11.614593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.614096    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r" event={"ID":"fcd7f922-a1f9-4bc8-995b-fbd2991589d9","Type":"ContainerDied","Data":"38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3"}
Nov 03 06:10:11.614593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.614160    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38feb47d94bedd065dc941e71a50d5f853352dda61126dd51006c88c402fa8b3"
Nov 03 06:10:11.616560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.616534    4528 generic.go:356] "Generic (PLEG): container finished" podID="9c6dd176-5325-4ac3-b302-8c2cc615e7c6" containerID="9be43a0145d73021f5462bce2d538a29b857d90fdfa3d6399f0970f31e670dfd" exitCode=0
Nov 03 06:10:11.616705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.616576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7" event={"ID":"9c6dd176-5325-4ac3-b302-8c2cc615e7c6","Type":"ContainerDied","Data":"9be43a0145d73021f5462bce2d538a29b857d90fdfa3d6399f0970f31e670dfd"}
Nov 03 06:10:11.776867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.776769    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:10:11.795073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:11.795036    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:10:12.419069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.419026    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:12.821565 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.821539    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:12.921467 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.921433    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\") pod \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") "
Nov 03 06:10:12.922380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.922351    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\" (UniqueName: \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\") pod \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") "
Nov 03 06:10:12.922948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.922139    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca" pod "9c6dd176-5325-4ac3-b302-8c2cc615e7c6" (UID: "9c6dd176-5325-4ac3-b302-8c2cc615e7c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:12.923057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.922929    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\") pod \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\" (UID: \"9c6dd176-5325-4ac3-b302-8c2cc615e7c6\") "
Nov 03 06:10:12.923930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.923902    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:12.925277 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.925254    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2" pod "9c6dd176-5325-4ac3-b302-8c2cc615e7c6" (UID: "9c6dd176-5325-4ac3-b302-8c2cc615e7c6"). InnerVolumeSpecName "kube-api-access-jsvv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:12.929234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:12.929210    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca" pod "9c6dd176-5325-4ac3-b302-8c2cc615e7c6" (UID: "9c6dd176-5325-4ac3-b302-8c2cc615e7c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:13.025306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:13.025275    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsvv2\" (UniqueName: \"kubernetes.io/projected/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-kube-api-access-jsvv2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:13.025306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:13.025306    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c6dd176-5325-4ac3-b302-8c2cc615e7c6-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:13.659819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:13.655277    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7" event={"ID":"9c6dd176-5325-4ac3-b302-8c2cc615e7c6","Type":"ContainerDied","Data":"8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da"}
Nov 03 06:10:13.659819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:13.655347    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c138715bf8e4153aae47906f687f0f777c84dceac9f94b89d1171ecea4749da"
Nov 03 06:10:13.659819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:13.655422    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"
Nov 03 06:10:14.029890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:14.020520    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"]
Nov 03 06:10:14.053816 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:14.045472    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:10:14.181818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:14.179879    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-b7e447-jpxn4"]
Nov 03 06:10:14.947821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:14.942896    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.177819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.170169    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:10:15.177819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.171962    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded"
Nov 03 06:10:15.237023 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.236904    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.375102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.375046    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.411827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.411075    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.548877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.548034    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:10:15.567958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.559906    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.669819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.665891    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.903822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.889682    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d1c47c9-52ff-4861-9907-91a7596d2c90" path="/var/lib/kubelet/pods/4d1c47c9-52ff-4861-9907-91a7596d2c90/volumes"
Nov 03 06:10:15.945938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.945886    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:15.946980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:15.946953    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:16.215191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.215144    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.219817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.216413    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:10:16.245496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.245437    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.246284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.246260    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:10:16.286605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.286566    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.413080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.411053    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.413080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.411483    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.428360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.428327    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:10:16.428633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.428614    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:10:16.624382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.588484    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:16.624382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.589934    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:10:16.624382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.596923    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:10:16.624382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.619672    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:10:16.673849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.667510    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:10:16.673849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:16.668317    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:10:17.209819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.205950    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:17.232827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.232008    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.299816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.266868    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.416293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.414926    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:10:17.416293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.416051    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.515196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.515089    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.560897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.555679    4528 generic.go:356] "Generic (PLEG): container finished" podID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerID="f8f4ce8fee0507364f03470ac7d89c5479aec39c024fc847d1d4827d9bfdcea8" exitCode=0
Nov 03 06:10:17.560897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.555975    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" event={"ID":"d85b3cc0-9251-4c7c-b47a-55338091aba3","Type":"ContainerDied","Data":"f8f4ce8fee0507364f03470ac7d89c5479aec39c024fc847d1d4827d9bfdcea8"}
Nov 03 06:10:17.586877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.573433    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:17.586877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.573502    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.596319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.596272    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:17.597400 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.597371    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:10:17.712824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:17.712759    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.209753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.208906    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:18.236319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.236272    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.275211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.273520    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.317344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.317307    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:18.317571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.317542    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.366958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.366916    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:18.367170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.367138    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.425498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.425441    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.425953 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.425912    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:10:18.480872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.480727    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:10:18.517396 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.517347    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:18.517685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.517652    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.661822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.657885    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\") pod \"d85b3cc0-9251-4c7c-b47a-55338091aba3\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") "
Nov 03 06:10:18.702852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.665965    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\" (UniqueName: \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\") pod \"d85b3cc0-9251-4c7c-b47a-55338091aba3\" (UID: \"d85b3cc0-9251-4c7c-b47a-55338091aba3\") "
Nov 03 06:10:18.736880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.732986    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.762061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.678585    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:18.767824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.678624    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": context deadline exceeded"
Nov 03 06:10:18.792844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:18.664033    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume" pod "d85b3cc0-9251-4c7c-b47a-55338091aba3" (UID: "d85b3cc0-9251-4c7c-b47a-55338091aba3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:19.029084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.028981    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz" pod "d85b3cc0-9251-4c7c-b47a-55338091aba3" (UID: "d85b3cc0-9251-4c7c-b47a-55338091aba3"). InnerVolumeSpecName "kube-api-access-nk8nz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:19.061821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.036431    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d85b3cc0-9251-4c7c-b47a-55338091aba3-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:19.066819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.063181    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nk8nz\" (UniqueName: \"kubernetes.io/projected/d85b3cc0-9251-4c7c-b47a-55338091aba3-kube-api-access-nk8nz\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:19.094983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.059253    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf" event={"ID":"d85b3cc0-9251-4c7c-b47a-55338091aba3","Type":"ContainerDied","Data":"6dd6dbd2c4599261f4b998f064726906e868bfff3a8eb6909b2f73b70a153856"}
Nov 03 06:10:19.095390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.095361    4528 scope.go:122] "RemoveContainer" containerID="f8f4ce8fee0507364f03470ac7d89c5479aec39c024fc847d1d4827d9bfdcea8"
Nov 03 06:10:19.095878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.039601    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:19.120821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.097221    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.106:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.120821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.095900    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"
Nov 03 06:10:19.155821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.151959    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:19.180816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.156081    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.122:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.181235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.181110    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:19.181394 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.181369    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.112:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.227820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.215952    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:19.227820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.216046    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.227820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.216157    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.406867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.406770    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.407783 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.407575    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.432889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.432839    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:10:19.434587 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.434558    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.765133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.765078    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.767008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.766952    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:19.767579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:19.767237    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.168822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.145358    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:10:20.179126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.179077    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-b7e447-7f6cbb675-475rf"]
Nov 03 06:10:20.224816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.224049    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:20.454876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.440494    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.456656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.445949    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.457452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.445999    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.457858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.446029    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.793870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.791168    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:20.793870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.792897    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:10:20.799749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.780785    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:20.876151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:20.876097    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:21.255262 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.255215    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:21.365317 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.365280    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"]
Nov 03 06:10:21.368546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.368514    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9c6dd176-5325-4ac3-b302-8c2cc615e7c6" containerName="revoke-keys"
Nov 03 06:10:21.368752 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.368732    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c6dd176-5325-4ac3-b302-8c2cc615e7c6" containerName="revoke-keys"
Nov 03 06:10:21.368887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.368872    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fcd7f922-a1f9-4bc8-995b-fbd2991589d9" containerName="revoke-keys"
Nov 03 06:10:21.368985 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.368973    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcd7f922-a1f9-4bc8-995b-fbd2991589d9" containerName="revoke-keys"
Nov 03 06:10:21.369111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.369099    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing"
Nov 03 06:10:21.369203 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.369190    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing"
Nov 03 06:10:21.369490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.369473    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c6dd176-5325-4ac3-b302-8c2cc615e7c6" containerName="revoke-keys"
Nov 03 06:10:21.369596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.369584    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcd7f922-a1f9-4bc8-995b-fbd2991589d9" containerName="revoke-keys"
Nov 03 06:10:21.369694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.369683    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" containerName="payload-pre-processing"
Nov 03 06:10:21.433859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.424969    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.457725    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.457874    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.458217    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.458482    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded"
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.459104    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.459209    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:10:21.460112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.459473    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:21.498986 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.498927    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"]
Nov 03 06:10:21.653819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.611611    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:10:21.657828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.656020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5gg5\" (UniqueName: \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:21.657828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.656372    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:21.657828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.656525    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:21.813906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.809248    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:10:21.814250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.814202    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:21.814728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:21.814698    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:22.031875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.024361    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:22.031875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.024505    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5gg5\" (UniqueName: \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:22.087805 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.041074    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:22.107341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.089400    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:22.213819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.212755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:22.255952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.255897    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:22.466228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.466168    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:22.487606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.467868    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:22.489439 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.467922    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:10:22.498922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.498885    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:22.499158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.499127    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:22.499278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.468430    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d85b3cc0-9251-4c7c-b47a-55338091aba3" path="/var/lib/kubelet/pods/d85b3cc0-9251-4c7c-b47a-55338091aba3/volumes"
Nov 03 06:10:22.506821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.499991    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:10:22.507671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.467943    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.008833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:22.945248    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:23.033950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.033890    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.236817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.023787    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" probeResult="failure" output="Get \"https://10.132.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.236817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.023739    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" probeResult="failure" output="Get \"https://10.132.0.91:8000/ready\": context deadline exceeded"
Nov 03 06:10:23.254640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.163612    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:10:23.367816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.316914    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5gg5\" (UniqueName: \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\") pod \"maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:23.504054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.504004    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.689575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.689459    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:10:23.693333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.693282    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/ready\": context deadline exceeded"
Nov 03 06:10:23.707249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.707209    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:23.707571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.707526    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.708642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.708605    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:23.778314 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.559030    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:23.860000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.836975    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:23.887573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.887531    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 03 06:10:23.907461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:23.907406    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.100:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:24.114346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.098510    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:24.114346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.098568    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:24.148166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.148128    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:10:24.148340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.148254    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:10:24.148340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.148311    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:10:24.148340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.148327    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"]
Nov 03 06:10:24.164771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164727    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:10:24.164771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164765    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"]
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164828    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164852    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164899    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164913    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164956    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.164986    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:10:24.165032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.165014    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:24.167009 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.166129    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy" containerID="cri-o://fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e" gracePeriod=30
Nov 03 06:10:24.167009 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.166237    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy" containerID="cri-o://fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302" gracePeriod=30
Nov 03 06:10:24.168256 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.167480    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.175033 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.175013    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:10:24.175384 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.175355    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy" containerID="cri-o://478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed" gracePeriod=30
Nov 03 06:10:24.183950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.183872    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:10:24.184093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.184075    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy" containerID="cri-o://b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91" gracePeriod=30
Nov 03 06:10:24.194406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.194373    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:10:24.295665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.288375    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.295665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.288516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cfst\" (UniqueName: \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.295665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.288641    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.394474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.394414    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.394868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.394844    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.394990 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.394894    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cfst\" (UniqueName: \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.395653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.395624    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.398264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.398233    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.442136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.442105    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cfst\" (UniqueName: \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\") pod \"maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.500483 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.500390    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:24.531403 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:24.531180    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4dcb714_551f_4330_9546_d0fa4e1b8628.slice/crio-e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33 WatchSource:0}: Error finding container e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33: Status 404 returned error can't find the container with id e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33
Nov 03 06:10:24.564334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.564291    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"]
Nov 03 06:10:24.691259 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.691213    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4" event={"ID":"f4dcb714-551f-4330-9546-d0fa4e1b8628","Type":"ContainerStarted","Data":"e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33"}
Nov 03 06:10:24.862066 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:24.862029    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dba591a_cce5_4d60_a572_c5de8209e9b2.slice/crio-07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392 WatchSource:0}: Error finding container 07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392: Status 404 returned error can't find the container with id 07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392
Nov 03 06:10:24.863933 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.863862    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"]
Nov 03 06:10:24.872579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.872532    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"]
Nov 03 06:10:24.882843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.879323    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:24.907291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.906048    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:24.907291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.906248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6znz\" (UniqueName: \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:24.907291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.906381    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:24.911700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.910130    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"]
Nov 03 06:10:24.925909 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:24.925868    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:10:25.007774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.007732    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.007981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.007953    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6znz\" (UniqueName: \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.008151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.008134    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.008772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.008749    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.077695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.077658    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.078121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.078094    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6znz\" (UniqueName: \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\") pod \"maas-api-revoke-sub-b29e034a-6sfg4\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.220803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.220768    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:25.551863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.551714    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"]
Nov 03 06:10:25.705105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.705059    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4" event={"ID":"3dba591a-cce5-4d60-a572-c5de8209e9b2","Type":"ContainerStarted","Data":"652bd1b5af0fdc5aed6c0d1e36c3486a81922eb9794cbd9e74c36254340035c0"}
Nov 03 06:10:25.705308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.705132    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4" event={"ID":"3dba591a-cce5-4d60-a572-c5de8209e9b2","Type":"ContainerStarted","Data":"07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392"}
Nov 03 06:10:25.707324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.707293    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4" event={"ID":"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2","Type":"ContainerStarted","Data":"bebe8fcfce0290f74e056a7988a5dd188a9b7f621ee32d959753c7b65ea6b912"}
Nov 03 06:10:25.709546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.709515    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4" event={"ID":"f4dcb714-551f-4330-9546-d0fa4e1b8628","Type":"ContainerStarted","Data":"c82cc01859153af0a29d1e872f97296072983530e788897ac7488e385bca2dfa"}
Nov 03 06:10:25.750852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.750772    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4" podStartSLOduration=3.750757209 podStartE2EDuration="3.750757209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:25.744309619 +0000 UTC m=+914.155706698" watchObservedRunningTime="2026-11-03 06:10:25.750757209 +0000 UTC m=+914.162154669"
Nov 03 06:10:25.790172 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:25.790076    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4" podStartSLOduration=5.790055121 podStartE2EDuration="5.790055121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:25.787192842 +0000 UTC m=+914.198589949" watchObservedRunningTime="2026-11-03 06:10:25.790055121 +0000 UTC m=+914.201452201"
Nov 03 06:10:26.733624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.733588    4528 generic.go:356] "Generic (PLEG): container finished" podID="3dba591a-cce5-4d60-a572-c5de8209e9b2" containerID="652bd1b5af0fdc5aed6c0d1e36c3486a81922eb9794cbd9e74c36254340035c0" exitCode=0
Nov 03 06:10:26.734084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.733676    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4" event={"ID":"3dba591a-cce5-4d60-a572-c5de8209e9b2","Type":"ContainerDied","Data":"652bd1b5af0fdc5aed6c0d1e36c3486a81922eb9794cbd9e74c36254340035c0"}
Nov 03 06:10:26.735860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.735826    4528 generic.go:356] "Generic (PLEG): container finished" podID="e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" containerID="4a726ac5e467897dec5cfa0223961ad16fa6151e1238ec2b0d7c88f873672860" exitCode=0
Nov 03 06:10:26.735986 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.735909    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4" event={"ID":"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2","Type":"ContainerDied","Data":"4a726ac5e467897dec5cfa0223961ad16fa6151e1238ec2b0d7c88f873672860"}
Nov 03 06:10:26.737773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.737749    4528 generic.go:356] "Generic (PLEG): container finished" podID="f4dcb714-551f-4330-9546-d0fa4e1b8628" containerID="c82cc01859153af0a29d1e872f97296072983530e788897ac7488e385bca2dfa" exitCode=0
Nov 03 06:10:26.737929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.737775    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4" event={"ID":"f4dcb714-551f-4330-9546-d0fa4e1b8628","Type":"ContainerDied","Data":"c82cc01859153af0a29d1e872f97296072983530e788897ac7488e385bca2dfa"}
Nov 03 06:10:26.800032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.799984    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main" containerID="cri-o://b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762" gracePeriod=45
Nov 03 06:10:26.818247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:26.818199    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main" containerID="cri-o://15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26" gracePeriod=45
Nov 03 06:10:27.055345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.054729    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"]
Nov 03 06:10:27.062291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.062257    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.119020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.118888    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"]
Nov 03 06:10:27.145104 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.145066    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqpn\" (UniqueName: \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.145321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.145297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.145487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.145472    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.171353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.171316    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:10:27.179434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.179360    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:10:27.246499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246466    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.246681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246629    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.246681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246673    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.246810 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246775    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\" (UniqueName: \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.246876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246862    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\" (UniqueName: \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.246939 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246893    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\" (UniqueName: \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.246991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246936    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.246991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.246977    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.247089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247001    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.247089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247044    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.247089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247083    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.247240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247115    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\" (UniqueName: \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\") pod \"b7089559-f797-4ce8-bab2-828a446aee78\" (UID: \"b7089559-f797-4ce8-bab2-828a446aee78\") "
Nov 03 06:10:27.247240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247190    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\") pod \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\" (UID: \"00f97b2e-cdd4-46f8-a4ec-a573c06e794c\") "
Nov 03 06:10:27.247451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247422    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.247583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqpn\" (UniqueName: \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.247646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247578    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.247699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247660    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.247853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247830    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.248016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.247990    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.248111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248088    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.248169 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248122    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.248341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248317    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.248341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248341    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.248561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248356    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.248561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248368    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.248561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248380    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.248561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.248424    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.249343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.249313    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.251173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.251146    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.252272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.252245    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:27.252476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.252452    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.253876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.253836    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:27.253973 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.253914    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:27.254687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.254661    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6" pod "b7089559-f797-4ce8-bab2-828a446aee78" (UID: "b7089559-f797-4ce8-bab2-828a446aee78"). InnerVolumeSpecName "kube-api-access-qrrw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:27.256814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.255196    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c" pod "00f97b2e-cdd4-46f8-a4ec-a573c06e794c" (UID: "00f97b2e-cdd4-46f8-a4ec-a573c06e794c"). InnerVolumeSpecName "kube-api-access-lgw8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:27.287830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.286622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqpn\" (UniqueName: \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\") pod \"maas-api-revoke-sub-f23d1a4f-gk2hp\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.349912    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.349948    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgw8c\" (UniqueName: \"kubernetes.io/projected/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-kube-api-access-lgw8c\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.349963    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7089559-f797-4ce8-bab2-828a446aee78-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.349976    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00f97b2e-cdd4-46f8-a4ec-a573c06e794c-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.349989    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7089559-f797-4ce8-bab2-828a446aee78-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.350349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.350001    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrrw6\" (UniqueName: \"kubernetes.io/projected/b7089559-f797-4ce8-bab2-828a446aee78-kube-api-access-qrrw6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:27.414785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.414740    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:27.507510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.507323    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerName="payload-processing" containerID="cri-o://35efed43dbc1686a1a210db591b7f4639ef69447d23c7d67b58aba3cb8a9220e" gracePeriod=25
Nov 03 06:10:27.747626 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:27.745477    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bd2e4c5_6507_44c2_ad7e_066fc230ff69.slice/crio-dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a WatchSource:0}: Error finding container dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a: Status 404 returned error can't find the container with id dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a
Nov 03 06:10:27.751418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.751391    4528 generic.go:356] "Generic (PLEG): container finished" podID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerID="b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762" exitCode=0
Nov 03 06:10:27.751560 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.751513    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"
Nov 03 06:10:27.756601 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.756570    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" event={"ID":"00f97b2e-cdd4-46f8-a4ec-a573c06e794c","Type":"ContainerDied","Data":"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"}
Nov 03 06:10:27.756743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.756622    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"]
Nov 03 06:10:27.756743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.756664    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn" event={"ID":"00f97b2e-cdd4-46f8-a4ec-a573c06e794c","Type":"ContainerDied","Data":"7c1be09e5dc9acf40edf6d57efee4ae9c609b0fc42708173f94c9c4ecccb66c5"}
Nov 03 06:10:27.756743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.756688    4528 scope.go:122] "RemoveContainer" containerID="b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"
Nov 03 06:10:27.790036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.784375    4528 generic.go:356] "Generic (PLEG): container finished" podID="b7089559-f797-4ce8-bab2-828a446aee78" containerID="15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26" exitCode=0
Nov 03 06:10:27.790036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.784745    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" event={"ID":"b7089559-f797-4ce8-bab2-828a446aee78","Type":"ContainerDied","Data":"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"}
Nov 03 06:10:27.790036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.784788    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw" event={"ID":"b7089559-f797-4ce8-bab2-828a446aee78","Type":"ContainerDied","Data":"31580a815f714aa12c2e327750aa997936375738bc11e14690de66e597be762b"}
Nov 03 06:10:27.790036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.786084    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"
Nov 03 06:10:27.790523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.790488    4528 generic.go:356] "Generic (PLEG): container finished" podID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerID="35efed43dbc1686a1a210db591b7f4639ef69447d23c7d67b58aba3cb8a9220e" exitCode=0
Nov 03 06:10:27.790660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.790558    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" event={"ID":"b6ed1cc4-f120-43d1-8774-65bf446c151e","Type":"ContainerDied","Data":"35efed43dbc1686a1a210db591b7f4639ef69447d23c7d67b58aba3cb8a9220e"}
Nov 03 06:10:27.809639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.809611    4528 scope.go:122] "RemoveContainer" containerID="b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"
Nov 03 06:10:27.811209 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:27.810443    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762\": container with ID starting with b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762 not found: ID does not exist" containerID="b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"
Nov 03 06:10:27.811209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.810470    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762"} err="failed to get container status \"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762\": rpc error: code = NotFound desc = could not find container \"b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762\": container with ID starting with b3810113a3e30dbcf6079094acb59d22cc62e1903f1e68425f77f6d735cbe762 not found: ID does not exist"
Nov 03 06:10:27.811209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.810489    4528 scope.go:122] "RemoveContainer" containerID="15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"
Nov 03 06:10:27.828769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.828736    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:10:27.836849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.836758    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:10:27.838169 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.838139    4528 scope.go:122] "RemoveContainer" containerID="15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"
Nov 03 06:10:27.838425 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:27.838404    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26\": container with ID starting with 15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26 not found: ID does not exist" containerID="15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"
Nov 03 06:10:27.838519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.838437    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26"} err="failed to get container status \"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26\": rpc error: code = NotFound desc = could not find container \"15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26\": container with ID starting with 15b907a9e30527905454d2588284ef74956eea8e4f2587043bb20a74dc2b4b26 not found: ID does not exist"
Nov 03 06:10:27.847691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.847659    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/distinct-w5-ba6892-kserve-fb9f786bd-ffrbn"]
Nov 03 06:10:27.885609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.884718    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:10:27.897930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.897849    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-ba6892/simulator-w5-ba6892-kserve-54cd6d6c8d-q8pcw"]
Nov 03 06:10:27.976907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.976166    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\" (UniqueName: \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\") pod \"b6ed1cc4-f120-43d1-8774-65bf446c151e\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") "
Nov 03 06:10:27.976907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.976508    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\") pod \"b6ed1cc4-f120-43d1-8774-65bf446c151e\" (UID: \"b6ed1cc4-f120-43d1-8774-65bf446c151e\") "
Nov 03 06:10:27.977542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:27.977512    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume" pod "b6ed1cc4-f120-43d1-8774-65bf446c151e" (UID: "b6ed1cc4-f120-43d1-8774-65bf446c151e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:28.008594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.008552    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6" pod "b6ed1cc4-f120-43d1-8774-65bf446c151e" (UID: "b6ed1cc4-f120-43d1-8774-65bf446c151e"). InnerVolumeSpecName "kube-api-access-hnlj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:28.025650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.025620    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:28.077952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.077911    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\") pod \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") "
Nov 03 06:10:28.080737 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.080715    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:28.081303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.081280    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\" (UniqueName: \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\") pod \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") "
Nov 03 06:10:28.081481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.081464    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\") pod \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\" (UID: \"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2\") "
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.081950    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca" pod "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" (UID: "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.088617    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca" pod "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" (UID: "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.090248    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz" pod "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" (UID: "e2e1b35a-af1c-49da-b330-3ee39a7ad4a2"). InnerVolumeSpecName "kube-api-access-n6znz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.090924    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlj6\" (UniqueName: \"kubernetes.io/projected/b6ed1cc4-f120-43d1-8774-65bf446c151e-kube-api-access-hnlj6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.090948    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.090963    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6ed1cc4-f120-43d1-8774-65bf446c151e-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.091859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.090979    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.108360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.108333    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:28.192479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192443    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\") pod \"3dba591a-cce5-4d60-a572-c5de8209e9b2\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") "
Nov 03 06:10:28.192630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192532    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\") pod \"f4dcb714-551f-4330-9546-d0fa4e1b8628\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") "
Nov 03 06:10:28.192630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192585    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\") pod \"f4dcb714-551f-4330-9546-d0fa4e1b8628\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") "
Nov 03 06:10:28.192630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192621    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\" (UniqueName: \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\") pod \"3dba591a-cce5-4d60-a572-c5de8209e9b2\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") "
Nov 03 06:10:28.192864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192677    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\" (UniqueName: \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\") pod \"f4dcb714-551f-4330-9546-d0fa4e1b8628\" (UID: \"f4dcb714-551f-4330-9546-d0fa4e1b8628\") "
Nov 03 06:10:28.192966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192946    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\") pod \"3dba591a-cce5-4d60-a572-c5de8209e9b2\" (UID: \"3dba591a-cce5-4d60-a572-c5de8209e9b2\") "
Nov 03 06:10:28.193023 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.192961    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca" pod "f4dcb714-551f-4330-9546-d0fa4e1b8628" (UID: "f4dcb714-551f-4330-9546-d0fa4e1b8628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:28.193750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.193726    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.193920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.193905    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6znz\" (UniqueName: \"kubernetes.io/projected/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2-kube-api-access-n6znz\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.194533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.194507    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca" pod "3dba591a-cce5-4d60-a572-c5de8209e9b2" (UID: "3dba591a-cce5-4d60-a572-c5de8209e9b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:28.195817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.195498    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst" pod "3dba591a-cce5-4d60-a572-c5de8209e9b2" (UID: "3dba591a-cce5-4d60-a572-c5de8209e9b2"). InnerVolumeSpecName "kube-api-access-4cfst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:28.195817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.195747    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5" pod "f4dcb714-551f-4330-9546-d0fa4e1b8628" (UID: "f4dcb714-551f-4330-9546-d0fa4e1b8628"). InnerVolumeSpecName "kube-api-access-k5gg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:28.196139 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.196115    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca" pod "f4dcb714-551f-4330-9546-d0fa4e1b8628" (UID: "f4dcb714-551f-4330-9546-d0fa4e1b8628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:28.197090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.197065    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca" pod "3dba591a-cce5-4d60-a572-c5de8209e9b2" (UID: "3dba591a-cce5-4d60-a572-c5de8209e9b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:28.296204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.296143    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.296204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.296186    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dba591a-cce5-4d60-a572-c5de8209e9b2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.296204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.296201    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4dcb714-551f-4330-9546-d0fa4e1b8628-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.296408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.296214    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cfst\" (UniqueName: \"kubernetes.io/projected/3dba591a-cce5-4d60-a572-c5de8209e9b2-kube-api-access-4cfst\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.296408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.296228    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5gg5\" (UniqueName: \"kubernetes.io/projected/f4dcb714-551f-4330-9546-d0fa4e1b8628-kube-api-access-k5gg5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:28.815462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.815145    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4" event={"ID":"f4dcb714-551f-4330-9546-d0fa4e1b8628","Type":"ContainerDied","Data":"e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33"}
Nov 03 06:10:28.815462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.815260    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e8c8dd159118bbc94be8f5d83b29c915a808e9f4cc3bce1afd72e2a286c38e33"
Nov 03 06:10:28.815462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.815367    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"
Nov 03 06:10:28.818160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.818126    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl" event={"ID":"b6ed1cc4-f120-43d1-8774-65bf446c151e","Type":"ContainerDied","Data":"65a73fbb92731d225ce52f3578d9243feac270325fd47dca3de65f097eab6493"}
Nov 03 06:10:28.818383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.818364    4528 scope.go:122] "RemoveContainer" containerID="35efed43dbc1686a1a210db591b7f4639ef69447d23c7d67b58aba3cb8a9220e"
Nov 03 06:10:28.818676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.818657    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"
Nov 03 06:10:28.824462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.824429    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp" event={"ID":"2bd2e4c5-6507-44c2-ad7e-066fc230ff69","Type":"ContainerStarted","Data":"8aae7550d30764f3cbf374776c76ab0c077bb270c946e01914efd511f95932bd"}
Nov 03 06:10:28.824575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.824496    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp" event={"ID":"2bd2e4c5-6507-44c2-ad7e-066fc230ff69","Type":"ContainerStarted","Data":"dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a"}
Nov 03 06:10:28.828474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.828449    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4" event={"ID":"3dba591a-cce5-4d60-a572-c5de8209e9b2","Type":"ContainerDied","Data":"07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392"}
Nov 03 06:10:28.828619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.828603    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07c1c3425c22cdc163dc44efd052bd16125a6f5e95aafa920d539ef454c0b392"
Nov 03 06:10:28.828837 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.828696    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"
Nov 03 06:10:28.835087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.835062    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4" event={"ID":"e2e1b35a-af1c-49da-b330-3ee39a7ad4a2","Type":"ContainerDied","Data":"bebe8fcfce0290f74e056a7988a5dd188a9b7f621ee32d959753c7b65ea6b912"}
Nov 03 06:10:28.835161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.835082    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"
Nov 03 06:10:28.835161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.835110    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bebe8fcfce0290f74e056a7988a5dd188a9b7f621ee32d959753c7b65ea6b912"
Nov 03 06:10:28.884868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.884781    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:10:28.891441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.891409    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-ba6892-b7bb697df-56qxl"]
Nov 03 06:10:28.939621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:28.939560    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp" podStartSLOduration=1.93954249 podStartE2EDuration="1.93954249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:28.932814915 +0000 UTC m=+917.344211990" watchObservedRunningTime="2026-11-03 06:10:28.93954249 +0000 UTC m=+917.350939569"
Nov 03 06:10:29.151769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.151655    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing" containerID="cri-o://5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d" gracePeriod=25
Nov 03 06:10:29.612693 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.612278    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:10:29.664426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.664402    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:10:29.670069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.670044    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:10:29.674386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.674362    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:10:29.741314 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.741277    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.741825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.741781    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.741938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.741906    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742044    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742103    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742132    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742175    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742270    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742893 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742443    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\") pod \"18544b11-2328-4148-b9a8-2533f1fc4822\" (UID: \"18544b11-2328-4148-b9a8-2533f1fc4822\") "
Nov 03 06:10:29.742893 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742680    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.743016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.742950    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.743154 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743123    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.743306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743284    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.743390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743305    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.743390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743319    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.743390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743303    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.743640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.743574    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:29.745720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.745698    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "kube-api-access-htkf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.746103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.746084    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.746207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.746179    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.747969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.747945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo" pod "18544b11-2328-4148-b9a8-2533f1fc4822" (UID: "18544b11-2328-4148-b9a8-2533f1fc4822"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:29.760808 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.759100    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" path="/var/lib/kubelet/pods/00f97b2e-cdd4-46f8-a4ec-a573c06e794c/volumes"
Nov 03 06:10:29.760808 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.759716    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" path="/var/lib/kubelet/pods/b6ed1cc4-f120-43d1-8774-65bf446c151e/volumes"
Nov 03 06:10:29.760808 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.760265    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7089559-f797-4ce8-bab2-828a446aee78" path="/var/lib/kubelet/pods/b7089559-f797-4ce8-bab2-828a446aee78/volumes"
Nov 03 06:10:29.844685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.844649    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.845294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845269    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.845479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845460    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.845624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.845775 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845759    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.845941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845923    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846067    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.846223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846207    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.845055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:29.846421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846329    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.846421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846408    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846452    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846558    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.846630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846596    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.846715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846697    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.846771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846741    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846773    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.846850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846826    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846859    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.846952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846895    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.846952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846938    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.847102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.846974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.847102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.847046    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.847204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.847111    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.847204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.847156    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\") pod \"ba872758-9e9a-451f-b3c7-1e9563c31336\" (UID: \"ba872758-9e9a-451f-b3c7-1e9563c31336\") "
Nov 03 06:10:29.847335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.847311    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.848054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.847762    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:29.848499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.848227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.849107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.849083    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.850539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.850273    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.850539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.850499    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.850749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.850731    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.853511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.853105    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "kube-api-access-7m2pn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.853511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.853320    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.853680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.853554    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.853741 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.853722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.854162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.853891    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854316    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854410    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\") pod \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\" (UID: \"6e73abb8-51c6-406a-87d6-f090c3e5ac26\") "
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854447    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854486    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854498    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\") pod \"a019eebc-2834-4c31-b4df-b484cf5c4708\" (UID: \"a019eebc-2834-4c31-b4df-b484cf5c4708\") "
Nov 03 06:10:29.854680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.854647    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:29.855056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855032    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:29.855269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855249    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855545    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855573    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855596    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855610    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18544b11-2328-4148-b9a8-2533f1fc4822-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855623    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkf8\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-kube-api-access-htkf8\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855636    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855649    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7m2pn\" (UniqueName: \"kubernetes.io/projected/ba872758-9e9a-451f-b3c7-1e9563c31336-kube-api-access-7m2pn\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855658    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855667    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855675    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855686    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855698    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855710    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855718    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18544b11-2328-4148-b9a8-2533f1fc4822-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855725    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855734    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18544b11-2328-4148-b9a8-2533f1fc4822-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855743    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855751    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855762    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a019eebc-2834-4c31-b4df-b484cf5c4708-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.855770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855774    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855720    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855834    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18544b11-2328-4148-b9a8-2533f1fc4822-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855870    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855886    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba872758-9e9a-451f-b3c7-1e9563c31336-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855906    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.855918    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856298    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856353    4528 generic.go:356] "Generic (PLEG): container finished" podID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerID="b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91" exitCode=0
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856403    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856435    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" event={"ID":"ba872758-9e9a-451f-b3c7-1e9563c31336","Type":"ContainerDied","Data":"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"}
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856484    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8" event={"ID":"ba872758-9e9a-451f-b3c7-1e9563c31336","Type":"ContainerDied","Data":"61aa9649ae569219a53fb9fe2d8f281f73e7626401fad255b3c48bcb4a9d358f"}
Nov 03 06:10:29.857116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.856514    4528 scope.go:122] "RemoveContainer" containerID="b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"
Nov 03 06:10:29.857684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.857162    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:29.857684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.857197    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.857684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.857273    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:29.857684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.857277    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7" pod "6e73abb8-51c6-406a-87d6-f090c3e5ac26" (UID: "6e73abb8-51c6-406a-87d6-f090c3e5ac26"). InnerVolumeSpecName "kube-api-access-flrt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.857684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.857304    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn" pod "a019eebc-2834-4c31-b4df-b484cf5c4708" (UID: "a019eebc-2834-4c31-b4df-b484cf5c4708"). InnerVolumeSpecName "kube-api-access-trtcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:29.858640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.858617    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo" pod "ba872758-9e9a-451f-b3c7-1e9563c31336" (UID: "ba872758-9e9a-451f-b3c7-1e9563c31336"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:29.858971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.858948    4528 generic.go:356] "Generic (PLEG): container finished" podID="2bd2e4c5-6507-44c2-ad7e-066fc230ff69" containerID="8aae7550d30764f3cbf374776c76ab0c077bb270c946e01914efd511f95932bd" exitCode=0
Nov 03 06:10:29.859171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.859149    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp" event={"ID":"2bd2e4c5-6507-44c2-ad7e-066fc230ff69","Type":"ContainerDied","Data":"8aae7550d30764f3cbf374776c76ab0c077bb270c946e01914efd511f95932bd"}
Nov 03 06:10:29.860947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.860924    4528 generic.go:356] "Generic (PLEG): container finished" podID="18544b11-2328-4148-b9a8-2533f1fc4822" containerID="fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e" exitCode=0
Nov 03 06:10:29.861028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.860984    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"
Nov 03 06:10:29.861028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.861002    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" event={"ID":"18544b11-2328-4148-b9a8-2533f1fc4822","Type":"ContainerDied","Data":"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"}
Nov 03 06:10:29.861119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.861058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg" event={"ID":"18544b11-2328-4148-b9a8-2533f1fc4822","Type":"ContainerDied","Data":"2787f492276f7f7f8b0f1b16484bb4b849a082991af7d0cf969d534a2c613018"}
Nov 03 06:10:29.864545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.863894    4528 generic.go:356] "Generic (PLEG): container finished" podID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerID="478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed" exitCode=0
Nov 03 06:10:29.864545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.863927    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"
Nov 03 06:10:29.864545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.863970    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerDied","Data":"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"}
Nov 03 06:10:29.864545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.864044    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w" event={"ID":"6e73abb8-51c6-406a-87d6-f090c3e5ac26","Type":"ContainerDied","Data":"75eafb22fad5c6e8aa1fc46a0bb2953f506ecde76be399893dfc05a8845f35ea"}
Nov 03 06:10:29.865599 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.865578    4528 generic.go:356] "Generic (PLEG): container finished" podID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerID="fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302" exitCode=0
Nov 03 06:10:29.865677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.865623    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"
Nov 03 06:10:29.865677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.865646    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" event={"ID":"a019eebc-2834-4c31-b4df-b484cf5c4708","Type":"ContainerDied","Data":"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"}
Nov 03 06:10:29.865740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.865683    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx" event={"ID":"a019eebc-2834-4c31-b4df-b484cf5c4708","Type":"ContainerDied","Data":"0181fa94ed5b2232dbe00d86abe3c15be9973d8373f1a29a3628742ac404d5a0"}
Nov 03 06:10:29.873685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.873658    4528 scope.go:122] "RemoveContainer" containerID="b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"
Nov 03 06:10:29.874274 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:29.874123    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91\": container with ID starting with b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91 not found: ID does not exist" containerID="b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"
Nov 03 06:10:29.874274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.874156    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91"} err="failed to get container status \"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91\": rpc error: code = NotFound desc = could not find container \"b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91\": container with ID starting with b984fa7ade622a179eb621da212632940741841b7eb51dbf1742a87e29f5fe91 not found: ID does not exist"
Nov 03 06:10:29.874274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.874189    4528 scope.go:122] "RemoveContainer" containerID="fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"
Nov 03 06:10:29.887318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.887294    4528 scope.go:122] "RemoveContainer" containerID="fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"
Nov 03 06:10:29.887668 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:29.887646    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e\": container with ID starting with fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e not found: ID does not exist" containerID="fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"
Nov 03 06:10:29.887756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.887677    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e"} err="failed to get container status \"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e\": rpc error: code = NotFound desc = could not find container \"fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e\": container with ID starting with fcc0c727444cd65d13d0d9bb661c443da39db3b0038f8666f6ce1d08df9df98e not found: ID does not exist"
Nov 03 06:10:29.887756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.887700    4528 scope.go:122] "RemoveContainer" containerID="478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"
Nov 03 06:10:29.898677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.898658    4528 scope.go:122] "RemoveContainer" containerID="f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"
Nov 03 06:10:29.909230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.909209    4528 scope.go:122] "RemoveContainer" containerID="478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"
Nov 03 06:10:29.909496 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:29.909479    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed\": container with ID starting with 478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed not found: ID does not exist" containerID="478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"
Nov 03 06:10:29.909586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.909506    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed"} err="failed to get container status \"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed\": rpc error: code = NotFound desc = could not find container \"478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed\": container with ID starting with 478cbf18a3e88ec1b62d512bcd54392879fd142282bfab6aed87cd0c521e78ed not found: ID does not exist"
Nov 03 06:10:29.909586 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.909525    4528 scope.go:122] "RemoveContainer" containerID="f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"
Nov 03 06:10:29.909828 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:29.909780    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352\": container with ID starting with f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352 not found: ID does not exist" containerID="f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"
Nov 03 06:10:29.909828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.909817    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352"} err="failed to get container status \"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352\": rpc error: code = NotFound desc = could not find container \"f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352\": container with ID starting with f5e318aea21cbf9907677d1626a0bb0718abcc4dbf36d07c278849b086fc4352 not found: ID does not exist"
Nov 03 06:10:29.909937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.909835    4528 scope.go:122] "RemoveContainer" containerID="fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"
Nov 03 06:10:29.920050 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.920032    4528 scope.go:122] "RemoveContainer" containerID="fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"
Nov 03 06:10:29.920373 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:29.920352    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302\": container with ID starting with fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302 not found: ID does not exist" containerID="fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"
Nov 03 06:10:29.920445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.920379    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302"} err="failed to get container status \"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302\": rpc error: code = NotFound desc = could not find container \"fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302\": container with ID starting with fc03b6d56e0a41e70da1323d7c519da6435c73fe3738fee7161ae5adea3d9302 not found: ID does not exist"
Nov 03 06:10:29.940515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.940490    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:10:29.947106 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.947076    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-fzfgg"]
Nov 03 06:10:29.957700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957673    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrt7\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-kube-api-access-flrt7\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957696    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957707    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trtcn\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-kube-api-access-trtcn\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957721    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957733    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957740    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a019eebc-2834-4c31-b4df-b484cf5c4708-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957747    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba872758-9e9a-451f-b3c7-1e9563c31336-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957754    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6e73abb8-51c6-406a-87d6-f090c3e5ac26-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:29.957818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:29.957762    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6e73abb8-51c6-406a-87d6-f090c3e5ac26-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:30.024220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.024170    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:10:30.030760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.030731    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-kdx4w"]
Nov 03 06:10:30.078412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.078362    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:10:30.079405 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.079377    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-s4rmx"]
Nov 03 06:10:30.214757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.214727    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:10:30.225293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:30.225261    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-b7e447-openshift-default-7c59b58bd5-vppf8"]
Nov 03 06:10:31.050948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.050918    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:31.071871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.071844    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\" (UniqueName: \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\") pod \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") "
Nov 03 06:10:31.072026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.071887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\") pod \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") "
Nov 03 06:10:31.072026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.071914    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\") pod \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\" (UID: \"2bd2e4c5-6507-44c2-ad7e-066fc230ff69\") "
Nov 03 06:10:31.072275 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.072251    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca" pod "2bd2e4c5-6507-44c2-ad7e-066fc230ff69" (UID: "2bd2e4c5-6507-44c2-ad7e-066fc230ff69"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:31.072504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.072487    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:31.074049 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.074028    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca" pod "2bd2e4c5-6507-44c2-ad7e-066fc230ff69" (UID: "2bd2e4c5-6507-44c2-ad7e-066fc230ff69"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:31.074108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.074071    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn" pod "2bd2e4c5-6507-44c2-ad7e-066fc230ff69" (UID: "2bd2e4c5-6507-44c2-ad7e-066fc230ff69"). InnerVolumeSpecName "kube-api-access-tcqpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:31.174242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.174146    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqpn\" (UniqueName: \"kubernetes.io/projected/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-kube-api-access-tcqpn\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:31.174242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.174178    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2bd2e4c5-6507-44c2-ad7e-066fc230ff69-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:31.411294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.411253    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:31.873857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.867838    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" path="/var/lib/kubelet/pods/18544b11-2328-4148-b9a8-2533f1fc4822/volumes"
Nov 03 06:10:31.873857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.868568    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" path="/var/lib/kubelet/pods/6e73abb8-51c6-406a-87d6-f090c3e5ac26/volumes"
Nov 03 06:10:31.876191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.876158    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" path="/var/lib/kubelet/pods/a019eebc-2834-4c31-b4df-b484cf5c4708/volumes"
Nov 03 06:10:31.877068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.877040    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" path="/var/lib/kubelet/pods/ba872758-9e9a-451f-b3c7-1e9563c31336/volumes"
Nov 03 06:10:31.907494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.907460    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"
Nov 03 06:10:31.907647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.907467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp" event={"ID":"2bd2e4c5-6507-44c2-ad7e-066fc230ff69","Type":"ContainerDied","Data":"dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a"}
Nov 03 06:10:31.907647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:31.907640    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc04b375990e7a9aca8f349b4f3fa0f39e7a5bf08c74140eb8127e9559be1b5a"
Nov 03 06:10:32.676572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:32.676534    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:10:36.249829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.249776    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"]
Nov 03 06:10:36.250775 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250751    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerName="payload-processing"
Nov 03 06:10:36.250840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250779    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerName="payload-processing"
Nov 03 06:10:36.250840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250811    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy"
Nov 03 06:10:36.250840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250819    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy"
Nov 03 06:10:36.250840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250836    4528 cpu_manager.go:414] "containerMap: removing container" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250843    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250856    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250863    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250881    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f4dcb714-551f-4330-9546-d0fa4e1b8628" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250888    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4dcb714-551f-4330-9546-d0fa4e1b8628" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250897    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250904    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250936    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2bd2e4c5-6507-44c2-ad7e-066fc230ff69" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250943    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2bd2e4c5-6507-44c2-ad7e-066fc230ff69" containerName="revoke-keys"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250956    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250963    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250975    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy"
Nov 03 06:10:36.250983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.250981    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251006    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251013    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251026    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3dba591a-cce5-4d60-a572-c5de8209e9b2" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251032    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dba591a-cce5-4d60-a572-c5de8209e9b2" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251047    4528 cpu_manager.go:414] "containerMap: removing container" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251054    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251212    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6ed1cc4-f120-43d1-8774-65bf446c151e" containerName="payload-processing"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251223    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba872758-9e9a-451f-b3c7-1e9563c31336" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251234    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="18544b11-2328-4148-b9a8-2533f1fc4822" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251243    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251252    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4dcb714-551f-4330-9546-d0fa4e1b8628" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251260    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251271    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2bd2e4c5-6507-44c2-ad7e-066fc230ff69" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251282    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7089559-f797-4ce8-bab2-828a446aee78" containerName="main"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251290    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a019eebc-2834-4c31-b4df-b484cf5c4708" containerName="istio-proxy"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251301    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dba591a-cce5-4d60-a572-c5de8209e9b2" containerName="revoke-keys"
Nov 03 06:10:36.251338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.251309    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="00f97b2e-cdd4-46f8-a4ec-a573c06e794c" containerName="main"
Nov 03 06:10:36.257291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.257269    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.275982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.275948    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"]
Nov 03 06:10:36.282095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.282065    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:10:36.282248 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.282186    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:10:36.282959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.282937    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:10:36.289373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.289352    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:10:36.431321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.431279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.431601 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.431577    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6h6z\" (UniqueName: \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.431833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.431784    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.533685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.533600    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.533858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.533741    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.533907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.533886    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6h6z\" (UniqueName: \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.534365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.534346    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.536222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.536200    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.557404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.557371    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6h6z\" (UniqueName: \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\") pod \"maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.573387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.573358    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:36.927484 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:36.927446    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e59ff32_d3b9_4011_a8ae_2ccdfa4294f5.slice/crio-ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757 WatchSource:0}: Error finding container ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757: Status 404 returned error can't find the container with id ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757
Nov 03 06:10:36.929186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:36.929150    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"]
Nov 03 06:10:37.012816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.010878    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6" event={"ID":"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5","Type":"ContainerStarted","Data":"ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757"}
Nov 03 06:10:37.399667 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.399625    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"]
Nov 03 06:10:37.400958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.400931    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e73abb8-51c6-406a-87d6-f090c3e5ac26" containerName="istio-proxy"
Nov 03 06:10:37.405770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.405745    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.437816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.435441    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"]
Nov 03 06:10:37.482040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.481924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.482367 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.482140    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsr6b\" (UniqueName: \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.482367 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.482178    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.521846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.519696    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"]
Nov 03 06:10:37.542157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.542121    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.552989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.552949    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"]
Nov 03 06:10:37.583562 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583531    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.583719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsr6b\" (UniqueName: \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.583719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583635    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.583870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583823    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwhq4\" (UniqueName: \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.583870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583858    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.583975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.583923    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.585962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.585933    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.603341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.603305    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.607369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.607341    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsr6b\" (UniqueName: \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\") pod \"maas-api-revoke-sub-6c5a406c-npw7f\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.677889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.677841    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing" containerID="cri-o://63d47b1da8aa96de9ccd0911e6713aaa6949d1535492714d6293a83bb823519c" gracePeriod=25
Nov 03 06:10:37.685117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.685087    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwhq4\" (UniqueName: \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.685247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.685125    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.685303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.685287    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.686008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.685983    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.687629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.687608    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.713281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.713257    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwhq4\" (UniqueName: \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\") pod \"maas-api-revoke-sub-49ea6a2c-whdtz\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.747386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.747309    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:37.892509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.892473    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:37.983975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.976991    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"]
Nov 03 06:10:37.996111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:37.995998    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-ba6892-lfx6r"]
Nov 03 06:10:38.033657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:38.032642    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6" event={"ID":"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5","Type":"ContainerStarted","Data":"6df1fb033685356930368e4e7379fea510270d88ff2a69eb3fdacc53b28a9c67"}
Nov 03 06:10:38.092918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:38.092887    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"]
Nov 03 06:10:38.107321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:38.107264    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6" podStartSLOduration=2.107249232 podStartE2EDuration="2.107249232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:38.081756972 +0000 UTC m=+926.493154051" watchObservedRunningTime="2026-11-03 06:10:38.107249232 +0000 UTC m=+926.518646310"
Nov 03 06:10:38.212310 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:38.212280    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0576776c_c783_4fa8_b9ad_9dac7f8c64d2.slice/crio-7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13 WatchSource:0}: Error finding container 7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13: Status 404 returned error can't find the container with id 7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13
Nov 03 06:10:38.217600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:38.217572    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"]
Nov 03 06:10:39.044590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.044507    4528 generic.go:356] "Generic (PLEG): container finished" podID="5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" containerID="6df1fb033685356930368e4e7379fea510270d88ff2a69eb3fdacc53b28a9c67" exitCode=0
Nov 03 06:10:39.045024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.044589    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6" event={"ID":"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5","Type":"ContainerDied","Data":"6df1fb033685356930368e4e7379fea510270d88ff2a69eb3fdacc53b28a9c67"}
Nov 03 06:10:39.046271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.046247    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f" event={"ID":"17e7983a-9e49-4b04-8be1-a8ab81e780c1","Type":"ContainerStarted","Data":"74758de8f7be49e6b92d7b5fa78fe680661177d8744d24d5fded4c7181038790"}
Nov 03 06:10:39.046391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.046286    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f" event={"ID":"17e7983a-9e49-4b04-8be1-a8ab81e780c1","Type":"ContainerStarted","Data":"d947b5c8720c895b3a59dfee2c02280672e9eb170b632e1cecf57db4b5d31a1f"}
Nov 03 06:10:39.048005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.047979    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz" event={"ID":"0576776c-c783-4fa8-b9ad-9dac7f8c64d2","Type":"ContainerStarted","Data":"f5a55b4dbd963a44cee2810a3e9f239fb1bfa6b8ccac74e7eab360736ddfc15b"}
Nov 03 06:10:39.048078 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.048015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz" event={"ID":"0576776c-c783-4fa8-b9ad-9dac7f8c64d2","Type":"ContainerStarted","Data":"7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13"}
Nov 03 06:10:39.128924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.128869    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f" podStartSLOduration=2.128854355 podStartE2EDuration="2.128854355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:39.124900596 +0000 UTC m=+927.536297673" watchObservedRunningTime="2026-11-03 06:10:39.128854355 +0000 UTC m=+927.540251433"
Nov 03 06:10:39.166052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.166000    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz" podStartSLOduration=2.165984528 podStartE2EDuration="2.165984528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:39.16484219 +0000 UTC m=+927.576239270" watchObservedRunningTime="2026-11-03 06:10:39.165984528 +0000 UTC m=+927.577381605"
Nov 03 06:10:39.731844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.731808    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:10:39.748677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.748639    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.794681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.794402    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcd7f922-a1f9-4bc8-995b-fbd2991589d9" path="/var/lib/kubelet/pods/fcd7f922-a1f9-4bc8-995b-fbd2991589d9/volumes"
Nov 03 06:10:39.795589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.795561    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:10:39.816004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.815963    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main" containerID="cri-o://acb6887b86064bb0017e90f4c81d70e3c0ffa2b88069a4f205bf3b7e7e758936" gracePeriod=45
Nov 03 06:10:39.836742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.836710    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xv5b\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.836961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.836938    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.837177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.837143    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.837644 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.837610    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.838001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.837893    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.838150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.838037    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.838150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.838123    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.838934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.838197    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.839718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.839692    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.866191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.866149    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:10:39.876275 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.876245    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.900953 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.899121    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:10:39.941558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941519    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.941693 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941583    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.941693 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.941867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941738    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4crt9\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.941867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941769    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.942319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.942292    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.947866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.941819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.947969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.947955    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.948102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.948185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948125    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948143    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948161    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.948389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948218    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948434    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948538    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.948647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948629    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948787    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.948911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.948850    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.949064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.949038    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.949115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.949102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.949972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.949516    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:39.949972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.949587    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xv5b\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.951531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.951495    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.951659 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.951622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.962017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.961975    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main" containerID="cri-o://ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76" gracePeriod=45
Nov 03 06:10:39.970366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.970339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:39.975445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:39.975389    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xv5b\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\") pod \"e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") " pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.051869    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052009    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052065    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4crt9\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052238    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052314    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052404    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052466    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.052610    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.053228    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"]
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.053300    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.053561    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.053946    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.054142    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.060830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.055176    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.066767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.066715    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.068005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.067944    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.069973 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.069941    4528 generic.go:356] "Generic (PLEG): container finished" podID="17e7983a-9e49-4b04-8be1-a8ab81e780c1" containerID="74758de8f7be49e6b92d7b5fa78fe680661177d8744d24d5fded4c7181038790" exitCode=0
Nov 03 06:10:40.070085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.070030    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f" event={"ID":"17e7983a-9e49-4b04-8be1-a8ab81e780c1","Type":"ContainerDied","Data":"74758de8f7be49e6b92d7b5fa78fe680661177d8744d24d5fded4c7181038790"}
Nov 03 06:10:40.072832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.072400    4528 generic.go:356] "Generic (PLEG): container finished" podID="0576776c-c783-4fa8-b9ad-9dac7f8c64d2" containerID="f5a55b4dbd963a44cee2810a3e9f239fb1bfa6b8ccac74e7eab360736ddfc15b" exitCode=0
Nov 03 06:10:40.072832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.072659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz" event={"ID":"0576776c-c783-4fa8-b9ad-9dac7f8c64d2","Type":"ContainerDied","Data":"f5a55b4dbd963a44cee2810a3e9f239fb1bfa6b8ccac74e7eab360736ddfc15b"}
Nov 03 06:10:40.073376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.073354    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.074575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.074548    4528 generic.go:356] "Generic (PLEG): container finished" podID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerID="acb6887b86064bb0017e90f4c81d70e3c0ffa2b88069a4f205bf3b7e7e758936" exitCode=0
Nov 03 06:10:40.074915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.074895    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" event={"ID":"37e31898-dee0-4975-8b4d-8b8f851b223e","Type":"ContainerDied","Data":"acb6887b86064bb0017e90f4c81d70e3c0ffa2b88069a4f205bf3b7e7e758936"}
Nov 03 06:10:40.091618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.091570    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"]
Nov 03 06:10:40.107973 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.106933    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:40.113854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.110349    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4crt9\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.130201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.130139    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\") pod \"e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") " pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.139062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.138975    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:10:40.153871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.153829    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgr4q\" (UniqueName: \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.154545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.154213    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.154922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.154748    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.217441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.217178    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:40.256303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.256276    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\" (UniqueName: \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.256722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.256702    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.256861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.256747    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.257041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257020    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.257127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257110    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\" (UniqueName: \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.257203 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257181    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\") pod \"37e31898-dee0-4975-8b4d-8b8f851b223e\" (UID: \"37e31898-dee0-4975-8b4d-8b8f851b223e\") "
Nov 03 06:10:40.257422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257355    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.257588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257569    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.257661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257637    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.257884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257406    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.257997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.257979    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.258115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.258097    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgr4q\" (UniqueName: \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.258363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.258345    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.258478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.258367    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.258478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.258383    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.259166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.258572    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.259336 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.259307    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "kube-api-access-xwz9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:40.260263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.259907    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.261296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.261261    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs" pod "37e31898-dee0-4975-8b4d-8b8f851b223e" (UID: "37e31898-dee0-4975-8b4d-8b8f851b223e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:40.263249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.263208    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.287458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.287385    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgr4q\" (UniqueName: \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\") pod \"maas-api-revoke-sub-88477545-dmvql\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.360619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.360528    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37e31898-dee0-4975-8b4d-8b8f851b223e-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.360619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.360566    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwz9x\" (UniqueName: \"kubernetes.io/projected/37e31898-dee0-4975-8b4d-8b8f851b223e-kube-api-access-xwz9x\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.360619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.360581    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37e31898-dee0-4975-8b4d-8b8f851b223e-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.419746 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.419690    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:40.424364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.423845    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:10:40.428647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.428526    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:40.483830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.483621    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:10:40.493433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.492915    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.493433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.492999    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.493433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.493042    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.564593    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\" (UniqueName: \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.564671    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.564729    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.564874    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\") pod \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.564952    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.565119    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\" (UniqueName: \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\") pod \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.566491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\" (UniqueName: \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.566539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\") pod \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\" (UID: \"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf\") "
Nov 03 06:10:40.568404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.566655    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\") pod \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\" (UID: \"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5\") "
Nov 03 06:10:40.570840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.570216    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca" pod "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" (UID: "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:40.570840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.570445    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.570840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.570670    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.570840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.570803    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.572962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.572281    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca" pod "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" (UID: "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:40.574824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.574772    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.574931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.574876    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.574931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.574922    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:10:40.581192 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.579264    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:10:40.608239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.608179    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "kube-api-access-8v5m5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:40.608566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.608523    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:40.608747 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.608724    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm" pod "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" (UID: "ccfca8f4-b49c-4c85-9e65-72548e8a5cbf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:40.611374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.611337    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z" pod "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" (UID: "5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5"). InnerVolumeSpecName "kube-api-access-p6h6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669303    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8v5m5\" (UniqueName: \"kubernetes.io/projected/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-kube-api-access-8v5m5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669326    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669352    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669363    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669371    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669380    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6h6z\" (UniqueName: \"kubernetes.io/projected/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-kube-api-access-p6h6z\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669409    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669421    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.672994 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.669429    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:40.833275 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:40.833236    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"]
Nov 03 06:10:41.052652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.052618    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:10:41.062752 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.062718    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:10:41.096086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.096000    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql" event={"ID":"c1fc2a23-1289-4249-9241-52a294fcf009","Type":"ContainerStarted","Data":"d2dc412f0fade6ec874c6e45c0b7ed9dafe1e7056f4519c6f96bf0bef0a85477"}
Nov 03 06:10:41.110846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.107732    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6" event={"ID":"5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5","Type":"ContainerDied","Data":"ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757"}
Nov 03 06:10:41.110846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.107779    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec39867d85bd636a89f867d55173e19471585519fccc536a7eb8e12acb213757"
Nov 03 06:10:41.110846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.107860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"
Nov 03 06:10:41.115028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.114939    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" event={"ID":"c220ecdf-1400-4b8b-8df1-ff68f00d8748","Type":"ContainerStarted","Data":"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"}
Nov 03 06:10:41.115028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.114994    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" event={"ID":"c220ecdf-1400-4b8b-8df1-ff68f00d8748","Type":"ContainerStarted","Data":"8f046790ab0bcda00b088a7671eafc9d7beb204f920e1771852857c531d16f0a"}
Nov 03 06:10:41.117135 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.117103    4528 generic.go:356] "Generic (PLEG): container finished" podID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerID="ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76" exitCode=0
Nov 03 06:10:41.117246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.117153    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"
Nov 03 06:10:41.117246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.117177    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" event={"ID":"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf","Type":"ContainerDied","Data":"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"}
Nov 03 06:10:41.117246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.117211    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687" event={"ID":"ccfca8f4-b49c-4c85-9e65-72548e8a5cbf","Type":"ContainerDied","Data":"014628b4dd95e03b22378f278e95139e4ca7a6f50339b9a2a85dc0ed6a5f0a6c"}
Nov 03 06:10:41.117246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.117234    4528 scope.go:122] "RemoveContainer" containerID="ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"
Nov 03 06:10:41.120432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.120397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" event={"ID":"ea3d0c16-444e-495c-8208-b505170d2aef","Type":"ContainerStarted","Data":"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"}
Nov 03 06:10:41.120574 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.120438    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" event={"ID":"ea3d0c16-444e-495c-8208-b505170d2aef","Type":"ContainerStarted","Data":"c707b8d6fbe683d9fb7ddb949c43e390a24fb5aeac9ff0ad3678019a0179d8eb"}
Nov 03 06:10:41.123830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.123224    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"
Nov 03 06:10:41.125823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.125503    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn" event={"ID":"37e31898-dee0-4975-8b4d-8b8f851b223e","Type":"ContainerDied","Data":"2be26789a5d5d91f8cb5e5b1f889a646135460b0808f3328ff76ce2b9add7a34"}
Nov 03 06:10:41.140006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.139982    4528 scope.go:122] "RemoveContainer" containerID="ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"
Nov 03 06:10:41.149201 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:41.149034    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76\": container with ID starting with ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76 not found: ID does not exist" containerID="ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"
Nov 03 06:10:41.149201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.149072    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76"} err="failed to get container status \"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76\": rpc error: code = NotFound desc = could not find container \"ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76\": container with ID starting with ff5c690c7ed8c780807bfc3ee31696f64dc32205e4050a342db58564fa55ef76 not found: ID does not exist"
Nov 03 06:10:41.149201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.149090    4528 scope.go:122] "RemoveContainer" containerID="acb6887b86064bb0017e90f4c81d70e3c0ffa2b88069a4f205bf3b7e7e758936"
Nov 03 06:10:41.219821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.219701    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:41.221559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.221528    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:41.221679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.221588    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:41.246875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.246785    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podStartSLOduration=2.24676492 podStartE2EDuration="2.24676492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:41.238157783 +0000 UTC m=+929.649554861" watchObservedRunningTime="2026-11-03 06:10:41.24676492 +0000 UTC m=+929.658161999"
Nov 03 06:10:41.309654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.309593    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podStartSLOduration=2.309571798 podStartE2EDuration="2.309571798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:41.29876262 +0000 UTC m=+929.710159729" watchObservedRunningTime="2026-11-03 06:10:41.309571798 +0000 UTC m=+929.720968876"
Nov 03 06:10:41.336107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.336056    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"]
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.336927    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main"
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.336946    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main"
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.336959    4528 cpu_manager.go:414] "containerMap: removing container" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main"
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.336965    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main"
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.337001    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" containerName="revoke-keys"
Nov 03 06:10:41.337041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.337007    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" containerName="revoke-keys"
Nov 03 06:10:41.337397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.337151    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" containerName="revoke-keys"
Nov 03 06:10:41.337397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.337172    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" containerName="main"
Nov 03 06:10:41.337397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.337182    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" containerName="main"
Nov 03 06:10:41.342196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.342173    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.354445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.354409    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"]
Nov 03 06:10:41.365596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.365566    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:10:41.376269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.376241    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-86728b/simulator-w6-86728b-kserve-75d75d6bbb-w8687"]
Nov 03 06:10:41.385363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.385339    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:10:41.392593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.392568    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:41.396433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.396409    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-be1a66/distinct-w4-be1a66-kserve-798875cfd-bcjzn"]
Nov 03 06:10:41.407578 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.407556    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.407814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.407775    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.407889 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.407867    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2h8f\" (UniqueName: \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.409538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.409503    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:41.428306 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.428281    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:41.509584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.509534    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\") pod \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") "
Nov 03 06:10:41.509826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.509775    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\") pod \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") "
Nov 03 06:10:41.510242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510217    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\") pod \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") "
Nov 03 06:10:41.510434 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510418    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\") pod \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") "
Nov 03 06:10:41.510676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510655    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\" (UniqueName: \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\") pod \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\" (UID: \"0576776c-c783-4fa8-b9ad-9dac7f8c64d2\") "
Nov 03 06:10:41.510876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca" pod "17e7983a-9e49-4b04-8be1-a8ab81e780c1" (UID: "17e7983a-9e49-4b04-8be1-a8ab81e780c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:41.510876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510613    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca" pod "0576776c-c783-4fa8-b9ad-9dac7f8c64d2" (UID: "0576776c-c783-4fa8-b9ad-9dac7f8c64d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:41.511038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.510945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\" (UniqueName: \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\") pod \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\" (UID: \"17e7983a-9e49-4b04-8be1-a8ab81e780c1\") "
Nov 03 06:10:41.511246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.511221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.511611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.511589    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.511723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.511706    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2h8f\" (UniqueName: \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.511840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.511824    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.511900 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.511844    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.512662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.512636    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.514774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.514659    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca" pod "0576776c-c783-4fa8-b9ad-9dac7f8c64d2" (UID: "0576776c-c783-4fa8-b9ad-9dac7f8c64d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:41.514774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.514724    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca" pod "17e7983a-9e49-4b04-8be1-a8ab81e780c1" (UID: "17e7983a-9e49-4b04-8be1-a8ab81e780c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:41.514931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.514885    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4" pod "0576776c-c783-4fa8-b9ad-9dac7f8c64d2" (UID: "0576776c-c783-4fa8-b9ad-9dac7f8c64d2"). InnerVolumeSpecName "kube-api-access-qwhq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:41.516674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.516649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.517047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.517022    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b" pod "17e7983a-9e49-4b04-8be1-a8ab81e780c1" (UID: "17e7983a-9e49-4b04-8be1-a8ab81e780c1"). InnerVolumeSpecName "kube-api-access-nsr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:41.536683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.536622    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2h8f\" (UniqueName: \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\") pod \"maas-api-revoke-sub-934fe5f1-7wl5k\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.612897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.612805    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsr6b\" (UniqueName: \"kubernetes.io/projected/17e7983a-9e49-4b04-8be1-a8ab81e780c1-kube-api-access-nsr6b\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.612897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.612837    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.612897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.612847    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17e7983a-9e49-4b04-8be1-a8ab81e780c1-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.612897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.612855    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwhq4\" (UniqueName: \"kubernetes.io/projected/0576776c-c783-4fa8-b9ad-9dac7f8c64d2-kube-api-access-qwhq4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:41.755851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.748988    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:41.808857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.784957    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37e31898-dee0-4975-8b4d-8b8f851b223e" path="/var/lib/kubelet/pods/37e31898-dee0-4975-8b4d-8b8f851b223e/volumes"
Nov 03 06:10:41.808857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:41.785845    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccfca8f4-b49c-4c85-9e65-72548e8a5cbf" path="/var/lib/kubelet/pods/ccfca8f4-b49c-4c85-9e65-72548e8a5cbf/volumes"
Nov 03 06:10:42.122040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.121872    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:42.150150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.150097    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:42.150357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.150179    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:42.160677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.160645    4528 generic.go:356] "Generic (PLEG): container finished" podID="c1fc2a23-1289-4249-9241-52a294fcf009" containerID="b990aa1f89200b254d6e0a1da15c9b9e23ed6365b953b67e29a1fb57d1efbf4e" exitCode=0
Nov 03 06:10:42.160811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.160729    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql" event={"ID":"c1fc2a23-1289-4249-9241-52a294fcf009","Type":"ContainerDied","Data":"b990aa1f89200b254d6e0a1da15c9b9e23ed6365b953b67e29a1fb57d1efbf4e"}
Nov 03 06:10:42.162291 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.162267    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f" event={"ID":"17e7983a-9e49-4b04-8be1-a8ab81e780c1","Type":"ContainerDied","Data":"d947b5c8720c895b3a59dfee2c02280672e9eb170b632e1cecf57db4b5d31a1f"}
Nov 03 06:10:42.162334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.162308    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d947b5c8720c895b3a59dfee2c02280672e9eb170b632e1cecf57db4b5d31a1f"
Nov 03 06:10:42.162395 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.162378    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"
Nov 03 06:10:42.164954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.164928    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"
Nov 03 06:10:42.165065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.165015    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz" event={"ID":"0576776c-c783-4fa8-b9ad-9dac7f8c64d2","Type":"ContainerDied","Data":"7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13"}
Nov 03 06:10:42.165145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.165060    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d20f5e2e78e890910a862498c3a227bd36b1a3155d3b1abe6abef749ae66e13"
Nov 03 06:10:42.234547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.224695    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:42.234547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.224782    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:42.251970 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:42.251940    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88d224c3_fd32_40c9_84fe_3ec2e2a77e93.slice/crio-3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669 WatchSource:0}: Error finding container 3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669: Status 404 returned error can't find the container with id 3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669
Nov 03 06:10:42.259085 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:42.259060    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"]
Nov 03 06:10:43.117821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.117275    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:43.117821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.117339    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:43.204817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.201948    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k" event={"ID":"88d224c3-fd32-40c9-84fe-3ec2e2a77e93","Type":"ContainerStarted","Data":"10f9363668ce4bb735165aabf9b665dd01d22bb676d1efada0e53d255fbaf00a"}
Nov 03 06:10:43.204817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.202004    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k" event={"ID":"88d224c3-fd32-40c9-84fe-3ec2e2a77e93","Type":"ContainerStarted","Data":"3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669"}
Nov 03 06:10:43.220709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.219211    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:43.220709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.219265    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:43.253428 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.253357    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k" podStartSLOduration=2.253334315 podStartE2EDuration="2.253334315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:43.24375059 +0000 UTC m=+931.655147670" watchObservedRunningTime="2026-11-03 06:10:43.253334315 +0000 UTC m=+931.664731394"
Nov 03 06:10:43.754746 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:43.754305    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:10:43.757993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.757967    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:43.855167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.855134    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\" (UniqueName: \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\") pod \"c1fc2a23-1289-4249-9241-52a294fcf009\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") "
Nov 03 06:10:43.855344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.855216    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\") pod \"c1fc2a23-1289-4249-9241-52a294fcf009\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") "
Nov 03 06:10:43.855471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.855452    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\") pod \"c1fc2a23-1289-4249-9241-52a294fcf009\" (UID: \"c1fc2a23-1289-4249-9241-52a294fcf009\") "
Nov 03 06:10:43.856337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.856300    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca" pod "c1fc2a23-1289-4249-9241-52a294fcf009" (UID: "c1fc2a23-1289-4249-9241-52a294fcf009"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:43.860817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.860596    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q" pod "c1fc2a23-1289-4249-9241-52a294fcf009" (UID: "c1fc2a23-1289-4249-9241-52a294fcf009"). InnerVolumeSpecName "kube-api-access-jgr4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:43.861141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.861122    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca" pod "c1fc2a23-1289-4249-9241-52a294fcf009" (UID: "c1fc2a23-1289-4249-9241-52a294fcf009"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:43.958767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.958677    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:43.958997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.958975    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgr4q\" (UniqueName: \"kubernetes.io/projected/c1fc2a23-1289-4249-9241-52a294fcf009-kube-api-access-jgr4q\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:43.959104 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:43.959087    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1fc2a23-1289-4249-9241-52a294fcf009-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:44.111769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.111729    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:44.112064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.112035    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:44.226877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.222530    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:44.226877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.222594    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:44.264006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.263962    4528 generic.go:356] "Generic (PLEG): container finished" podID="88d224c3-fd32-40c9-84fe-3ec2e2a77e93" containerID="10f9363668ce4bb735165aabf9b665dd01d22bb676d1efada0e53d255fbaf00a" exitCode=0
Nov 03 06:10:44.264217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.264044    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k" event={"ID":"88d224c3-fd32-40c9-84fe-3ec2e2a77e93","Type":"ContainerDied","Data":"10f9363668ce4bb735165aabf9b665dd01d22bb676d1efada0e53d255fbaf00a"}
Nov 03 06:10:44.265894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.265875    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"
Nov 03 06:10:44.266002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.265906    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql" event={"ID":"c1fc2a23-1289-4249-9241-52a294fcf009","Type":"ContainerDied","Data":"d2dc412f0fade6ec874c6e45c0b7ed9dafe1e7056f4519c6f96bf0bef0a85477"}
Nov 03 06:10:44.266002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:44.265949    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2dc412f0fade6ec874c6e45c0b7ed9dafe1e7056f4519c6f96bf0bef0a85477"
Nov 03 06:10:45.111818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.111230    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:45.111818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.111297    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:45.225819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.225397    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:45.225819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.225462    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:45.464874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.464655    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:45.961772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.961733    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:45.970430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.970398    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:10:45.970658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:45.970631    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" containerID="cri-o://8012a8a6c59d673a8d519f51d7b201bb47f19a7e98a1979c4b4c85ed105ec234" gracePeriod=30
Nov 03 06:10:46.058828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.058255    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:10:46.058828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.058506    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" containerID="cri-o://ecb23241f9bbcd9cda15f66158fa5d1d8c308e7db5451e723d21a00efd0362be" gracePeriod=30
Nov 03 06:10:46.059202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.059174    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\") pod \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") "
Nov 03 06:10:46.059285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.059266    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\") pod \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") "
Nov 03 06:10:46.059530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.059507    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\" (UniqueName: \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\") pod \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\" (UID: \"88d224c3-fd32-40c9-84fe-3ec2e2a77e93\") "
Nov 03 06:10:46.061590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.061024    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca" pod "88d224c3-fd32-40c9-84fe-3ec2e2a77e93" (UID: "88d224c3-fd32-40c9-84fe-3ec2e2a77e93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:46.063959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.063887    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f" pod "88d224c3-fd32-40c9-84fe-3ec2e2a77e93" (UID: "88d224c3-fd32-40c9-84fe-3ec2e2a77e93"). InnerVolumeSpecName "kube-api-access-d2h8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:46.066402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.066367    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca" pod "88d224c3-fd32-40c9-84fe-3ec2e2a77e93" (UID: "88d224c3-fd32-40c9-84fe-3ec2e2a77e93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:46.068253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.068223    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:10:46.068561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.068465    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" containerID="cri-o://e86ee71739b61f7f13fdc4985dc96a3113aa25e56877145c05d4471fb1b3bddc" gracePeriod=30
Nov 03 06:10:46.234879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.176082    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:46.234879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.176124    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2h8f\" (UniqueName: \"kubernetes.io/projected/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-kube-api-access-d2h8f\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:46.234879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.176141    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88d224c3-fd32-40c9-84fe-3ec2e2a77e93-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:46.316862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.290033    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:46.316862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.290111    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:46.316862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.290033    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:46.316862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.290175    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:46.373910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.373645    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:10:46.374430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.374358    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" containerID="cri-o://b00103f56f6fa98af108eede2c7aebb79f4f4cef993c0b66b8b0513a922791d9" gracePeriod=30
Nov 03 06:10:46.454941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.451714    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k" event={"ID":"88d224c3-fd32-40c9-84fe-3ec2e2a77e93","Type":"ContainerDied","Data":"3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669"}
Nov 03 06:10:46.454941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.451782    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e03e914ba843b015c5f4f121e11982487a72166ac8cd7c43466d6bb6d0bc669"
Nov 03 06:10:46.454941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:46.451877    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"
Nov 03 06:10:47.029137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.029100    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:10:47.053220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.053187    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:10:47.114559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.114143    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:47.114559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.114202    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:47.249849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.247517    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:47.249849 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.247597    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:47.458674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.458635    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:10:47.458934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.458707    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:10:47.711785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.711689    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"]
Nov 03 06:10:47.715477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715444    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0576776c-c783-4fa8-b9ad-9dac7f8c64d2" containerName="revoke-keys"
Nov 03 06:10:47.715477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715474    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0576776c-c783-4fa8-b9ad-9dac7f8c64d2" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715507    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c1fc2a23-1289-4249-9241-52a294fcf009" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715514    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1fc2a23-1289-4249-9241-52a294fcf009" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715552    4528 cpu_manager.go:414] "containerMap: removing container" podUID="17e7983a-9e49-4b04-8be1-a8ab81e780c1" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715559    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17e7983a-9e49-4b04-8be1-a8ab81e780c1" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="88d224c3-fd32-40c9-84fe-3ec2e2a77e93" containerName="revoke-keys"
Nov 03 06:10:47.715714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715597    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d224c3-fd32-40c9-84fe-3ec2e2a77e93" containerName="revoke-keys"
Nov 03 06:10:47.716063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715762    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="17e7983a-9e49-4b04-8be1-a8ab81e780c1" containerName="revoke-keys"
Nov 03 06:10:47.716063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715780    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d224c3-fd32-40c9-84fe-3ec2e2a77e93" containerName="revoke-keys"
Nov 03 06:10:47.716063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715808    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0576776c-c783-4fa8-b9ad-9dac7f8c64d2" containerName="revoke-keys"
Nov 03 06:10:47.716063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.715825    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1fc2a23-1289-4249-9241-52a294fcf009" containerName="revoke-keys"
Nov 03 06:10:47.721100 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.721075    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.737000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.736968    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:10:47.738195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.737992    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:10:47.738195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.738090    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:10:47.799579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.767685    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"]
Nov 03 06:10:47.799579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.767747    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"]
Nov 03 06:10:47.799579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.783305    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.799579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.783481    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh4jn\" (UniqueName: \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.799579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.783550    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.841627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.838308    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"]
Nov 03 06:10:47.841627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.840129    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:10:47.841627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.840252    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885347    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885410    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885486    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4qk\" (UniqueName: \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885881    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh4jn\" (UniqueName: \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885925    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.887818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.885985    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.906816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.889404    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.914911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.907199    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:47.997187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.997095    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.997354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.997298    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:47.997419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.997389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4qk\" (UniqueName: \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:48.003812 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:47.998755    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:48.007830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.004688    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:10:48.007830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.004738    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:10:48.012811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.011865    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:48.027852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.027190    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh4jn\" (UniqueName: \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\") pod \"maas-api-revoke-sub-387aa54e-x7fmp\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:48.034595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.034533    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4qk\" (UniqueName: \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\") pod \"maas-api-revoke-sub-353d94d9-926nq\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:48.052721 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.052433    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:48.117833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.116291    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Nov 03 06:10:48.117833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.116352    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Nov 03 06:10:48.177080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.173330    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:48.221031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.220507    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Nov 03 06:10:48.221031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.220564    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Nov 03 06:10:48.457279 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:48.457241    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10d665a3_6269_42b2_b006_2d5e0d9329c3.slice/crio-e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b WatchSource:0}: Error finding container e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b: Status 404 returned error can't find the container with id e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b
Nov 03 06:10:48.458623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.458597    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"]
Nov 03 06:10:48.480984 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.480947    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp" event={"ID":"10d665a3-6269-42b2-b006-2d5e0d9329c3","Type":"ContainerStarted","Data":"e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b"}
Nov 03 06:10:48.561481 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:48.561240    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365f8ed1_62fc_47b1_8818_92c71dd473ef.slice/crio-7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6 WatchSource:0}: Error finding container 7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6: Status 404 returned error can't find the container with id 7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6
Nov 03 06:10:48.564907 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:48.564662    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"]
Nov 03 06:10:49.042860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.042781    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"]
Nov 03 06:10:49.048179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.048151    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.074141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.074109    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"]
Nov 03 06:10:49.164266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.164233    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:49.165237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.165125    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:49.166193 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.166137    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:10:49.232234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.232202    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:49.232491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.232464    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:49.233593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.233568    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:10:49.237948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.237926    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.238158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.238138    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npc77\" (UniqueName: \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.238245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.238227    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.342020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.341981    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npc77\" (UniqueName: \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.342215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.342156    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.343698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.343650    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.344002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.343978    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.347102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.347052    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.369588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.369498    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npc77\" (UniqueName: \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\") pod \"maas-api-revoke-sub-280cfe3e-zctlt\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.503654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.503616    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq" event={"ID":"365f8ed1-62fc-47b1-8818-92c71dd473ef","Type":"ContainerStarted","Data":"9357775e4f2ff911043a3f8993095da269687716fab1d7912ef1aa792b637fd7"}
Nov 03 06:10:49.503923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.503684    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq" event={"ID":"365f8ed1-62fc-47b1-8818-92c71dd473ef","Type":"ContainerStarted","Data":"7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6"}
Nov 03 06:10:49.505412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.505372    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp" event={"ID":"10d665a3-6269-42b2-b006-2d5e0d9329c3","Type":"ContainerStarted","Data":"f4c9743cc2ccfdfc8d6f83032baa6899444a76e8da8061eac751b16fac472496"}
Nov 03 06:10:49.506222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.506197    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:10:49.506958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.506937    4528 generic.go:356] "Generic (PLEG): container finished" podID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerID="5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d" exitCode=0
Nov 03 06:10:49.507041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.507012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" event={"ID":"b79d5a77-7809-456f-bf14-e8cf9fde05d7","Type":"ContainerDied","Data":"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"}
Nov 03 06:10:49.507094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.507063    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6" event={"ID":"b79d5a77-7809-456f-bf14-e8cf9fde05d7","Type":"ContainerDied","Data":"0df0c6b5261c0a02b907ca610381019827fd0344ef32ad1c8d63a2d80af58439"}
Nov 03 06:10:49.507128 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.507093    4528 scope.go:122] "RemoveContainer" containerID="5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"
Nov 03 06:10:49.518755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.518723    4528 scope.go:122] "RemoveContainer" containerID="5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"
Nov 03 06:10:49.519055 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:49.519035    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d\": container with ID starting with 5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d not found: ID does not exist" containerID="5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"
Nov 03 06:10:49.519143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.519067    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d"} err="failed to get container status \"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d\": rpc error: code = NotFound desc = could not find container \"5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d\": container with ID starting with 5a9d62dec7766ad48d7e73ea28a81324fa4de7650095a02f97077d96680ad94d not found: ID does not exist"
Nov 03 06:10:49.547605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.547577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\" (UniqueName: \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\") pod \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") "
Nov 03 06:10:49.547724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.547694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\") pod \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\" (UID: \"b79d5a77-7809-456f-bf14-e8cf9fde05d7\") "
Nov 03 06:10:49.548110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.548082    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume" pod "b79d5a77-7809-456f-bf14-e8cf9fde05d7" (UID: "b79d5a77-7809-456f-bf14-e8cf9fde05d7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:49.549323 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.549300    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b79d5a77-7809-456f-bf14-e8cf9fde05d7-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:49.550262 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.550238    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878" pod "b79d5a77-7809-456f-bf14-e8cf9fde05d7" (UID: "b79d5a77-7809-456f-bf14-e8cf9fde05d7"). InnerVolumeSpecName "kube-api-access-4g878". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:49.555731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.555689    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq" podStartSLOduration=2.555677043 podStartE2EDuration="2.555677043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:49.549116652 +0000 UTC m=+937.960513766" watchObservedRunningTime="2026-11-03 06:10:49.555677043 +0000 UTC m=+937.967074121"
Nov 03 06:10:49.651420 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.651376    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g878\" (UniqueName: \"kubernetes.io/projected/b79d5a77-7809-456f-bf14-e8cf9fde05d7-kube-api-access-4g878\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:49.667596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.667559    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:49.685419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.685365    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp" podStartSLOduration=2.685346646 podStartE2EDuration="2.685346646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:49.67219689 +0000 UTC m=+938.083593969" watchObservedRunningTime="2026-11-03 06:10:49.685346646 +0000 UTC m=+938.096743726"
Nov 03 06:10:49.994563 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:49.994515    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa4d04ba_d007_4030_985a_50c4cff22772.slice/crio-29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39 WatchSource:0}: Error finding container 29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39: Status 404 returned error can't find the container with id 29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39
Nov 03 06:10:49.996326 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:49.996264    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"]
Nov 03 06:10:50.520667 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.520633    4528 generic.go:356] "Generic (PLEG): container finished" podID="10d665a3-6269-42b2-b006-2d5e0d9329c3" containerID="f4c9743cc2ccfdfc8d6f83032baa6899444a76e8da8061eac751b16fac472496" exitCode=0
Nov 03 06:10:50.521207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.520717    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp" event={"ID":"10d665a3-6269-42b2-b006-2d5e0d9329c3","Type":"ContainerDied","Data":"f4c9743cc2ccfdfc8d6f83032baa6899444a76e8da8061eac751b16fac472496"}
Nov 03 06:10:50.522060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.522039    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"
Nov 03 06:10:50.524062 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.524037    4528 generic.go:356] "Generic (PLEG): container finished" podID="365f8ed1-62fc-47b1-8818-92c71dd473ef" containerID="9357775e4f2ff911043a3f8993095da269687716fab1d7912ef1aa792b637fd7" exitCode=0
Nov 03 06:10:50.524961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.524133    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq" event={"ID":"365f8ed1-62fc-47b1-8818-92c71dd473ef","Type":"ContainerDied","Data":"9357775e4f2ff911043a3f8993095da269687716fab1d7912ef1aa792b637fd7"}
Nov 03 06:10:50.526253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.526228    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt" event={"ID":"aa4d04ba-d007-4030-985a-50c4cff22772","Type":"ContainerStarted","Data":"3bdf32eb8784a8bc1d7213d94b3bc8f08998684fd0c41c8aa3a94c6fd9249abb"}
Nov 03 06:10:50.526360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.526274    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt" event={"ID":"aa4d04ba-d007-4030-985a-50c4cff22772","Type":"ContainerStarted","Data":"29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39"}
Nov 03 06:10:50.672934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.672854    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:10:50.680334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.680300    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-ba6892-6fccb6fbf5-l22l6"]
Nov 03 06:10:50.711867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:50.711807    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt" podStartSLOduration=1.711774796 podStartE2EDuration="1.711774796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:10:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:10:50.705806071 +0000 UTC m=+939.117203143" watchObservedRunningTime="2026-11-03 06:10:50.711774796 +0000 UTC m=+939.123171873"
Nov 03 06:10:51.884816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:51.856931    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body=
Nov 03 06:10:51.884816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:51.856996    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused"
Nov 03 06:10:51.961822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:51.961343    4528 generic.go:356] "Generic (PLEG): container finished" podID="f2e49f45-2645-437a-8b7e-516634115788" containerID="8012a8a6c59d673a8d519f51d7b201bb47f19a7e98a1979c4b4c85ed105ec234" exitCode=0
Nov 03 06:10:51.961822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:51.961499    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" event={"ID":"f2e49f45-2645-437a-8b7e-516634115788","Type":"ContainerDied","Data":"8012a8a6c59d673a8d519f51d7b201bb47f19a7e98a1979c4b4c85ed105ec234"}
Nov 03 06:10:52.024557 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.024517    4528 generic.go:356] "Generic (PLEG): container finished" podID="aa4d04ba-d007-4030-985a-50c4cff22772" containerID="3bdf32eb8784a8bc1d7213d94b3bc8f08998684fd0c41c8aa3a94c6fd9249abb" exitCode=0
Nov 03 06:10:52.048157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.048109    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing" containerID="cri-o://c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c" gracePeriod=25
Nov 03 06:10:52.062774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.062719    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing" containerID="cri-o://91125266ea9ffc17ff9ecee8b2ea9cba70b121644bbf1eaba7932fe58b4c0f40" gracePeriod=25
Nov 03 06:10:52.064370 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.064155    4528 patch_prober.go:29] interesting pod/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused" start-of-body=
Nov 03 06:10:52.064686 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.064553    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.109:15021/healthz/ready\": dial tcp 10.132.0.109:15021: connect: connection refused"
Nov 03 06:10:52.107142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.100282    4528 generic.go:356] "Generic (PLEG): container finished" podID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerID="e86ee71739b61f7f13fdc4985dc96a3113aa25e56877145c05d4471fb1b3bddc" exitCode=0
Nov 03 06:10:52.164845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.164778    4528 generic.go:356] "Generic (PLEG): container finished" podID="80326b77-b409-4f1a-9deb-145840baaa82" containerID="ecb23241f9bbcd9cda15f66158fa5d1d8c308e7db5451e723d21a00efd0362be" exitCode=0
Nov 03 06:10:52.178716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.178682    4528 generic.go:356] "Generic (PLEG): container finished" podID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerID="b00103f56f6fa98af108eede2c7aebb79f4f4cef993c0b66b8b0513a922791d9" exitCode=0
Nov 03 06:10:52.340826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.305149    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" path="/var/lib/kubelet/pods/b79d5a77-7809-456f-bf14-e8cf9fde05d7/volumes"
Nov 03 06:10:53.068224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.448867    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:53.068960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:52.868199    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:53.069175 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.069144    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:53.072937 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:53.071404    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.104s"
Nov 03 06:10:53.076882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.076851    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt" event={"ID":"aa4d04ba-d007-4030-985a-50c4cff22772","Type":"ContainerDied","Data":"3bdf32eb8784a8bc1d7213d94b3bc8f08998684fd0c41c8aa3a94c6fd9249abb"}
Nov 03 06:10:53.077097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.077076    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" event={"ID":"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3","Type":"ContainerDied","Data":"e86ee71739b61f7f13fdc4985dc96a3113aa25e56877145c05d4471fb1b3bddc"}
Nov 03 06:10:53.106819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.077314    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" event={"ID":"80326b77-b409-4f1a-9deb-145840baaa82","Type":"ContainerDied","Data":"ecb23241f9bbcd9cda15f66158fa5d1d8c308e7db5451e723d21a00efd0362be"}
Nov 03 06:10:53.255694 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.255643    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" event={"ID":"d0f64805-2cbe-4e03-8f34-2104a7f76cd0","Type":"ContainerDied","Data":"b00103f56f6fa98af108eede2c7aebb79f4f4cef993c0b66b8b0513a922791d9"}
Nov 03 06:10:53.969871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.933943    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:10:53.969871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:53.934009    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:10:54.096819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.096583    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:10:54.160220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.130984    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:54.246978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.246936    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:10:54.268902 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.263103    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:10:54.429107 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.398221    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:10:54.608821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.582911    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/health\": context deadline exceeded"
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.591763    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722435    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722491    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722527    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722625    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722659    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722728    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722781    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722852    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722940    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.722982    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723097    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723156    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723194    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\") pod \"80326b77-b409-4f1a-9deb-145840baaa82\" (UID: \"80326b77-b409-4f1a-9deb-145840baaa82\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723233    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723310    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.724854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.723369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:54.732008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.731964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:54.867895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.815355    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:54.895220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.895041    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:54.895810 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.895443    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:54.895810 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.895510    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Nov 03 06:10:54.940022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.897038    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:54.940022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:54.922731    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.276828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.273255    4528 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/health\": context deadline exceeded"
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.279247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.279323    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" probeResult="failure" output="Get \"https://10.132.0.98:8000/ready\": context deadline exceeded"
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.304195    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") pod \"f2e49f45-2645-437a-8b7e-516634115788\" (UID: \"f2e49f45-2645-437a-8b7e-516634115788\") "
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305115    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305143    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305158    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305173    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305187    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305200    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80326b77-b409-4f1a-9deb-145840baaa82-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:10:55.305282    4528 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f2e49f45-2645-437a-8b7e-516634115788/volumes/kubernetes.io~empty-dir/istio-data
Nov 03 06:10:55.330976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.305295    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.402457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.402411    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" probeResult="failure" output="Get \"https://10.132.0.119:8000/ready\": context deadline exceeded"
Nov 03 06:10:55.480831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.452901    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.483768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.483026    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.483768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.483059    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.483768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.483215    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:10:55.489457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.488900    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.489457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.489153    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.494881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.494553    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:10:55.510964 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.507774    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.550471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.550431    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:55.566641 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:55.563059    4528 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.813s"
Nov 03 06:10:55.566641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.563144    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:10:55.566641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.563324    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:10:55.566641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.563354    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:10:55.628577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.604091    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded"
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.629886    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.644052    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.646825    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.646890    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.646929    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.646974    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647001    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647045    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647116    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647193    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647235    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647291    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647438    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647477    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647534    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647582    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647649    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647696    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647746    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\") pod \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\" (UID: \"d0f64805-2cbe-4e03-8f34-2104a7f76cd0\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.647778    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\") pod \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\" (UID: \"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3\") "
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.648495    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.648518    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2e49f45-2645-437a-8b7e-516634115788-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.648533    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2e49f45-2645-437a-8b7e-516634115788-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.660954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.648547    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2e49f45-2645-437a-8b7e-516634115788-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.662444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.662101    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:10:55.666813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.666770    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:55.667249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.667227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.667669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.667647    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.670591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.669733    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.670591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.669999    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.670591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.670158    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.670591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.670348    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.677987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.677952    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:55.678274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.678247    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.678423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.678401    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "kube-api-access-psbn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.678572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.678557    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "kube-api-access-8dgh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.709029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.704327    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.709029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.704436    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.709029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.706769    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "kube-api-access-xxdcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.709785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.709726    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.711204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.711165    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.717326 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.714955    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.717326 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.715042    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:55.717326 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.715338    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:55.722246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.722217    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf" event={"ID":"9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3","Type":"ContainerDied","Data":"32feec40a3f812ad1c84b6a3abf5d3d5a1f9cfc57049547dec43978445e092d7"}
Nov 03 06:10:55.722463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.722445    4528 scope.go:122] "RemoveContainer" containerID="e86ee71739b61f7f13fdc4985dc96a3113aa25e56877145c05d4471fb1b3bddc"
Nov 03 06:10:55.722892 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.722873    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"
Nov 03 06:10:55.724422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.724404    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:55.728840 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.727723    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:55.737851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.733753    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj" event={"ID":"80326b77-b409-4f1a-9deb-145840baaa82","Type":"ContainerDied","Data":"a07396e578de25b4ac4a404d387598501e499c361c02ff33af9be8273ebf5ea0"}
Nov 03 06:10:55.749886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.733820    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.752968    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psbn7\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-kube-api-access-psbn7\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.752999    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753017    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdcc\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-kube-api-access-xxdcc\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753032    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753046    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753060    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753074    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753087    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgh2\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-kube-api-access-8dgh2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753104    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753118    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753131    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80326b77-b409-4f1a-9deb-145840baaa82-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753144    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753158    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80326b77-b409-4f1a-9deb-145840baaa82-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753172    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753185    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753199    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753214    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753227    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.754542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.753513    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:10:55.756910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.756878    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "kube-api-access-nvsgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.757015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.756976    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token" pod "f2e49f45-2645-437a-8b7e-516634115788" (UID: "f2e49f45-2645-437a-8b7e-516634115788"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.758460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.758407    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token" pod "80326b77-b409-4f1a-9deb-145840baaa82" (UID: "80326b77-b409-4f1a-9deb-145840baaa82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.758967 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.758875    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token" pod "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" (UID: "9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.760839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.760261    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"
Nov 03 06:10:55.771847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.766775    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token" pod "d0f64805-2cbe-4e03-8f34-2104a7f76cd0" (UID: "d0f64805-2cbe-4e03-8f34-2104a7f76cd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.795240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.794998    4528 scope.go:122] "RemoveContainer" containerID="ecb23241f9bbcd9cda15f66158fa5d1d8c308e7db5451e723d21a00efd0362be"
Nov 03 06:10:55.814596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.814564    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"
Nov 03 06:10:55.824913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.824551    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"
Nov 03 06:10:55.827460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.827433    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:55.827897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.827699    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq" event={"ID":"d0f64805-2cbe-4e03-8f34-2104a7f76cd0","Type":"ContainerDied","Data":"7321266670ef7bfb28243d516f224c7bbc52d94b665f0308155dc2466a3a5de8"}
Nov 03 06:10:55.828117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.828098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b" event={"ID":"f2e49f45-2645-437a-8b7e-516634115788","Type":"ContainerDied","Data":"21a269a59a7867a997cdc812f641ffd3e06acd32e8fd5076776c1ab5145e9dc7"}
Nov 03 06:10:55.828277 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.828263    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp" event={"ID":"10d665a3-6269-42b2-b006-2d5e0d9329c3","Type":"ContainerDied","Data":"e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b"}
Nov 03 06:10:55.828397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.828380    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e37aaed07dc5e5e6c8f9ddba1e1039ebd421cf131b8916e186f61c40233bc26b"
Nov 03 06:10:55.846402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.846380    4528 scope.go:122] "RemoveContainer" containerID="b00103f56f6fa98af108eede2c7aebb79f4f4cef993c0b66b8b0513a922791d9"
Nov 03 06:10:55.854038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854001    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\") pod \"10d665a3-6269-42b2-b006-2d5e0d9329c3\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") "
Nov 03 06:10:55.854185 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854162    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\") pod \"aa4d04ba-d007-4030-985a-50c4cff22772\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") "
Nov 03 06:10:55.854282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854261    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\" (UniqueName: \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\") pod \"aa4d04ba-d007-4030-985a-50c4cff22772\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") "
Nov 03 06:10:55.854359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854303    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\") pod \"aa4d04ba-d007-4030-985a-50c4cff22772\" (UID: \"aa4d04ba-d007-4030-985a-50c4cff22772\") "
Nov 03 06:10:55.854438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854421    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\") pod \"10d665a3-6269-42b2-b006-2d5e0d9329c3\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") "
Nov 03 06:10:55.854589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854562    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\" (UniqueName: \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\") pod \"10d665a3-6269-42b2-b006-2d5e0d9329c3\" (UID: \"10d665a3-6269-42b2-b006-2d5e0d9329c3\") "
Nov 03 06:10:55.855105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.854419    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca" pod "10d665a3-6269-42b2-b006-2d5e0d9329c3" (UID: "10d665a3-6269-42b2-b006-2d5e0d9329c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.855239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855218    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80326b77-b409-4f1a-9deb-145840baaa82-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855246    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2e49f45-2645-437a-8b7e-516634115788-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855260    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855273    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855287    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855301    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvsgh\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-kube-api-access-nvsgh\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.855566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.855340    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0f64805-2cbe-4e03-8f34-2104a7f76cd0-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.856338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.856312    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca" pod "aa4d04ba-d007-4030-985a-50c4cff22772" (UID: "aa4d04ba-d007-4030-985a-50c4cff22772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.871837 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.871645    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn" pod "10d665a3-6269-42b2-b006-2d5e0d9329c3" (UID: "10d665a3-6269-42b2-b006-2d5e0d9329c3"). InnerVolumeSpecName "kube-api-access-rh4jn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.873187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.873159    4528 scope.go:122] "RemoveContainer" containerID="8012a8a6c59d673a8d519f51d7b201bb47f19a7e98a1979c4b4c85ed105ec234"
Nov 03 06:10:55.876582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.876107    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca" pod "aa4d04ba-d007-4030-985a-50c4cff22772" (UID: "aa4d04ba-d007-4030-985a-50c4cff22772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:55.876582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.876281    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77" pod "aa4d04ba-d007-4030-985a-50c4cff22772" (UID: "aa4d04ba-d007-4030-985a-50c4cff22772"). InnerVolumeSpecName "kube-api-access-npc77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.879969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.879939    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca" pod "10d665a3-6269-42b2-b006-2d5e0d9329c3" (UID: "10d665a3-6269-42b2-b006-2d5e0d9329c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:55.955249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.955213    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:10:55.956440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.956416    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\" (UniqueName: \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\") pod \"365f8ed1-62fc-47b1-8818-92c71dd473ef\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") "
Nov 03 06:10:55.956636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.956616    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\") pod \"365f8ed1-62fc-47b1-8818-92c71dd473ef\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") "
Nov 03 06:10:55.956705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.956682    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\") pod \"365f8ed1-62fc-47b1-8818-92c71dd473ef\" (UID: \"365f8ed1-62fc-47b1-8818-92c71dd473ef\") "
Nov 03 06:10:55.957053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957020    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca" pod "365f8ed1-62fc-47b1-8818-92c71dd473ef" (UID: "365f8ed1-62fc-47b1-8818-92c71dd473ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:55.957534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957509    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10d665a3-6269-42b2-b006-2d5e0d9329c3-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.957631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957537    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.957631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957552    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh4jn\" (UniqueName: \"kubernetes.io/projected/10d665a3-6269-42b2-b006-2d5e0d9329c3-kube-api-access-rh4jn\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.957631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957564    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.957631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957578    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npc77\" (UniqueName: \"kubernetes.io/projected/aa4d04ba-d007-4030-985a-50c4cff22772-kube-api-access-npc77\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.957631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.957587    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa4d04ba-d007-4030-985a-50c4cff22772-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:55.958768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.958743    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk" pod "365f8ed1-62fc-47b1-8818-92c71dd473ef" (UID: "365f8ed1-62fc-47b1-8818-92c71dd473ef"). InnerVolumeSpecName "kube-api-access-9s4qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:55.959141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.959123    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca" pod "365f8ed1-62fc-47b1-8818-92c71dd473ef" (UID: "365f8ed1-62fc-47b1-8818-92c71dd473ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:55.962374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:55.962352    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-h5r4b"]
Nov 03 06:10:56.059061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.059004    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s4qk\" (UniqueName: \"kubernetes.io/projected/365f8ed1-62fc-47b1-8818-92c71dd473ef-kube-api-access-9s4qk\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.059061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.059058    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/365f8ed1-62fc-47b1-8818-92c71dd473ef-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.090663 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.090602    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main" containerID="cri-o://e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c" gracePeriod=45
Nov 03 06:10:56.095292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.095252    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main" containerID="cri-o://e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa" gracePeriod=45
Nov 03 06:10:56.226363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.226328    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:10:56.244926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.244894    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-6xhdq"]
Nov 03 06:10:56.296845 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.296334    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:56.297266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.296946    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:10:56.298414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.298392    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:10:56.310417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.310354    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-g92wj"]
Nov 03 06:10:56.336917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.336876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:10:56.347295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.347249    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-ba6892-openshift-default-8548d7cc5f-nmnqf"]
Nov 03 06:10:56.441778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.441749    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:56.444678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.444661    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:10:56.568410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568316    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.568410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568380    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.568638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568419    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.568638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568536    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.568638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568571    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.568638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568600    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.568886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568651    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.568886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568653    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.568886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568692    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.568886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568721    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.568886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568754    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\" (UniqueName: \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.569151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568886    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\" (UniqueName: \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\") pod \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\" (UID: \"ac8f80a9-d28a-4f4c-a0d8-13555623a606\") "
Nov 03 06:10:56.569151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568954    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\") pod \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\" (UID: \"a5b698bb-3674-490d-bbd2-62fbd4b7497e\") "
Nov 03 06:10:56.569151 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.568947    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569471    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569674    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569944    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569962    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569970    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.570168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.569977    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.570510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.570487    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.570702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.570656    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.571628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.571598    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "kube-api-access-5rvbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:56.571739 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.571719    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.571739 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.571728    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:56.571879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.571748    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:10:56.573334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.573314    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm" pod "a5b698bb-3674-490d-bbd2-62fbd4b7497e" (UID: "a5b698bb-3674-490d-bbd2-62fbd4b7497e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:10:56.574613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.574593    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz" pod "ac8f80a9-d28a-4f4c-a0d8-13555623a606" (UID: "ac8f80a9-d28a-4f4c-a0d8-13555623a606"). InnerVolumeSpecName "kube-api-access-82lsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673284    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673325    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673340    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac8f80a9-d28a-4f4c-a0d8-13555623a606-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673362    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac8f80a9-d28a-4f4c-a0d8-13555623a606-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673382    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5b698bb-3674-490d-bbd2-62fbd4b7497e-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673400    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5b698bb-3674-490d-bbd2-62fbd4b7497e-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673423    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rvbr\" (UniqueName: \"kubernetes.io/projected/a5b698bb-3674-490d-bbd2-62fbd4b7497e-kube-api-access-5rvbr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.673485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.673445    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82lsz\" (UniqueName: \"kubernetes.io/projected/ac8f80a9-d28a-4f4c-a0d8-13555623a606-kube-api-access-82lsz\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:56.837295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.837199    4528 generic.go:356] "Generic (PLEG): container finished" podID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerID="e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa" exitCode=0
Nov 03 06:10:56.837295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.837242    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"
Nov 03 06:10:56.837295 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.837269    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" event={"ID":"ac8f80a9-d28a-4f4c-a0d8-13555623a606","Type":"ContainerDied","Data":"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"}
Nov 03 06:10:56.837546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.837309    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk" event={"ID":"ac8f80a9-d28a-4f4c-a0d8-13555623a606","Type":"ContainerDied","Data":"1c15eb7b484946dc1917fb9ef2b6e14071aa6a969b36b244dd7ecf4dd68424e5"}
Nov 03 06:10:56.837546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.837329    4528 scope.go:122] "RemoveContainer" containerID="e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"
Nov 03 06:10:56.839348 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.839324    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq" event={"ID":"365f8ed1-62fc-47b1-8818-92c71dd473ef","Type":"ContainerDied","Data":"7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6"}
Nov 03 06:10:56.839477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.839372    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aac8a71376b1a2d87b02f82a5e6c9380917a3620251839922b160140e46dfb6"
Nov 03 06:10:56.839477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.839330    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"
Nov 03 06:10:56.842579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.842440    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"
Nov 03 06:10:56.842579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.842483    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt" event={"ID":"aa4d04ba-d007-4030-985a-50c4cff22772","Type":"ContainerDied","Data":"29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39"}
Nov 03 06:10:56.842579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.842533    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29edcc8239e15772deed3bbc72c922991e88cc022312f1a7e8a9d3b53e6e1c39"
Nov 03 06:10:56.844344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.844318    4528 generic.go:356] "Generic (PLEG): container finished" podID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerID="e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c" exitCode=0
Nov 03 06:10:56.844453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.844396    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" event={"ID":"a5b698bb-3674-490d-bbd2-62fbd4b7497e","Type":"ContainerDied","Data":"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"}
Nov 03 06:10:56.844453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.844424    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp" event={"ID":"a5b698bb-3674-490d-bbd2-62fbd4b7497e","Type":"ContainerDied","Data":"8203d26d0e27ff7c666df79978cc606395b0aa07cffc658857a72396b4b1be38"}
Nov 03 06:10:56.844569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.844478    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"
Nov 03 06:10:56.853631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.853417    4528 scope.go:122] "RemoveContainer" containerID="e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"
Nov 03 06:10:56.853993 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:56.853966    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa\": container with ID starting with e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa not found: ID does not exist" containerID="e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"
Nov 03 06:10:56.854063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.854000    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa"} err="failed to get container status \"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa\": rpc error: code = NotFound desc = could not find container \"e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa\": container with ID starting with e2f1b23b9ac094ec1c5a2114fefe4a04f024e8a6d3d9e9e87dd3aa6a561390fa not found: ID does not exist"
Nov 03 06:10:56.854063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.854025    4528 scope.go:122] "RemoveContainer" containerID="e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"
Nov 03 06:10:56.866181 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.866161    4528 scope.go:122] "RemoveContainer" containerID="e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"
Nov 03 06:10:56.866464 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:10:56.866443    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c\": container with ID starting with e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c not found: ID does not exist" containerID="e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"
Nov 03 06:10:56.866529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.866471    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c"} err="failed to get container status \"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c\": rpc error: code = NotFound desc = could not find container \"e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c\": container with ID starting with e31ad56fd5c30b9791ae4ee488cf904d745b5a0fb5d7408be5e4ad2192accc1c not found: ID does not exist"
Nov 03 06:10:56.907971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.907922    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:10:56.914650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.914617    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/premium-w0-df7236-kserve-7599955768-5gkhp"]
Nov 03 06:10:56.969016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.968977    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:10:56.985689 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:56.985655    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-df7236/simulator-w0-df7236-kserve-6fb95fcd4f-m5ptk"]
Nov 03 06:10:57.751159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.751126    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80326b77-b409-4f1a-9deb-145840baaa82" path="/var/lib/kubelet/pods/80326b77-b409-4f1a-9deb-145840baaa82/volumes"
Nov 03 06:10:57.751676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.751660    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" path="/var/lib/kubelet/pods/9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3/volumes"
Nov 03 06:10:57.752097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.752084    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" path="/var/lib/kubelet/pods/a5b698bb-3674-490d-bbd2-62fbd4b7497e/volumes"
Nov 03 06:10:57.752825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.752810    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" path="/var/lib/kubelet/pods/ac8f80a9-d28a-4f4c-a0d8-13555623a606/volumes"
Nov 03 06:10:57.753244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.753228    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" path="/var/lib/kubelet/pods/d0f64805-2cbe-4e03-8f34-2104a7f76cd0/volumes"
Nov 03 06:10:57.753651 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.753638    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2e49f45-2645-437a-8b7e-516634115788" path="/var/lib/kubelet/pods/f2e49f45-2645-437a-8b7e-516634115788/volumes"
Nov 03 06:10:57.871559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.871525    4528 generic.go:356] "Generic (PLEG): container finished" podID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerID="63d47b1da8aa96de9ccd0911e6713aaa6949d1535492714d6293a83bb823519c" exitCode=0
Nov 03 06:10:57.871709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.871627    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" event={"ID":"498de3a8-6ecf-40fa-af5a-31845d02d5a8","Type":"ContainerDied","Data":"63d47b1da8aa96de9ccd0911e6713aaa6949d1535492714d6293a83bb823519c"}
Nov 03 06:10:57.966671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:57.966642    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:10:58.111446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.111406    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\") pod \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") "
Nov 03 06:10:58.111623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.111517    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\" (UniqueName: \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\") pod \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\" (UID: \"498de3a8-6ecf-40fa-af5a-31845d02d5a8\") "
Nov 03 06:10:58.111829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.111776    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume" pod "498de3a8-6ecf-40fa-af5a-31845d02d5a8" (UID: "498de3a8-6ecf-40fa-af5a-31845d02d5a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:10:58.112054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.112038    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/498de3a8-6ecf-40fa-af5a-31845d02d5a8-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:58.113741 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.113718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc" pod "498de3a8-6ecf-40fa-af5a-31845d02d5a8" (UID: "498de3a8-6ecf-40fa-af5a-31845d02d5a8"). InnerVolumeSpecName "kube-api-access-svfpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:10:58.214101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.214009    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svfpc\" (UniqueName: \"kubernetes.io/projected/498de3a8-6ecf-40fa-af5a-31845d02d5a8-kube-api-access-svfpc\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:10:58.884019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.883989    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"
Nov 03 06:10:58.884548 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.883988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd" event={"ID":"498de3a8-6ecf-40fa-af5a-31845d02d5a8","Type":"ContainerDied","Data":"58fe25715ab70d2bfed1afed81df967c9566afa14f19695b6127e27d2d303544"}
Nov 03 06:10:58.884548 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.884129    4528 scope.go:122] "RemoveContainer" containerID="63d47b1da8aa96de9ccd0911e6713aaa6949d1535492714d6293a83bb823519c"
Nov 03 06:10:58.944357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.944318    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:10:58.970806 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:58.970769    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-86728b-7d9bf7d865-npvkd"]
Nov 03 06:10:59.750614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:10:59.750582    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" path="/var/lib/kubelet/pods/498de3a8-6ecf-40fa-af5a-31845d02d5a8/volumes"
Nov 03 06:11:01.099422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.099382    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"]
Nov 03 06:11:01.100432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100404    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main"
Nov 03 06:11:01.100432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100427    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100440    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100450    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100466    4528 cpu_manager.go:414] "containerMap: removing container" podUID="365f8ed1-62fc-47b1-8818-92c71dd473ef" containerName="revoke-keys"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100475    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365f8ed1-62fc-47b1-8818-92c71dd473ef" containerName="revoke-keys"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100494    4528 cpu_manager.go:414] "containerMap: removing container" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100502    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100519    4528 cpu_manager.go:414] "containerMap: removing container" podUID="10d665a3-6269-42b2-b006-2d5e0d9329c3" containerName="revoke-keys"
Nov 03 06:11:01.100535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100526    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10d665a3-6269-42b2-b006-2d5e0d9329c3" containerName="revoke-keys"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100539    4528 cpu_manager.go:414] "containerMap: removing container" podUID="aa4d04ba-d007-4030-985a-50c4cff22772" containerName="revoke-keys"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100547    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa4d04ba-d007-4030-985a-50c4cff22772" containerName="revoke-keys"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100562    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100569    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100597    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100609    4528 cpu_manager.go:414] "containerMap: removing container" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100616    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100628    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100635    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100654    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing"
Nov 03 06:11:01.100779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100662    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100810    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa4d04ba-d007-4030-985a-50c4cff22772" containerName="revoke-keys"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100827    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f64805-2cbe-4e03-8f34-2104a7f76cd0" containerName="istio-proxy"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100839    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2e49f45-2645-437a-8b7e-516634115788" containerName="istio-proxy"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100847    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="80326b77-b409-4f1a-9deb-145840baaa82" containerName="istio-proxy"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100856    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b2cff8e-1e9c-45d4-8ac1-34dfcb1cbcc3" containerName="istio-proxy"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100866    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b79d5a77-7809-456f-bf14-e8cf9fde05d7" containerName="payload-pre-processing"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100881    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac8f80a9-d28a-4f4c-a0d8-13555623a606" containerName="main"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100890    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5b698bb-3674-490d-bbd2-62fbd4b7497e" containerName="main"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100900    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="365f8ed1-62fc-47b1-8818-92c71dd473ef" containerName="revoke-keys"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100908    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="498de3a8-6ecf-40fa-af5a-31845d02d5a8" containerName="payload-processing"
Nov 03 06:11:01.101161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.100919    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="10d665a3-6269-42b2-b006-2d5e0d9329c3" containerName="revoke-keys"
Nov 03 06:11:01.107044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.107021    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.136700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.136663    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:11:01.136700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.136678    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:11:01.136955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.136716    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:11:01.137101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.137074    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:11:01.143506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.143479    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"]
Nov 03 06:11:01.257215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.257176    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv6sm\" (UniqueName: \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.257449 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.257427    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.257500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.257467    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.358637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.358538    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.358637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.358588    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.358637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.358633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv6sm\" (UniqueName: \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.359436 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.359414    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.361042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.361019    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.381688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.381661    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv6sm\" (UniqueName: \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\") pod \"maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.422809 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.422773    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:01.755645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.755622    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"]
Nov 03 06:11:01.934276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:01.934230    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt" event={"ID":"75569012-1c4e-45ea-a1f5-42d09d184a07","Type":"ContainerStarted","Data":"0bd6449471996b2f8d1e08d8e6c4f5d8bc9f33d2be8108307dae590a017785d5"}
Nov 03 06:11:02.697692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:02.697658    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"]
Nov 03 06:11:02.711117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:02.711081    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-be1a66-8ffs4"]
Nov 03 06:11:02.946634 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:02.946591    4528 generic.go:356] "Generic (PLEG): container finished" podID="75569012-1c4e-45ea-a1f5-42d09d184a07" containerID="07973c529ce482929bfc3ef559b47e1bf7c6b187c4e1ebb1f6ca67544a53ff9c" exitCode=0
Nov 03 06:11:02.946782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:02.946671    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt" event={"ID":"75569012-1c4e-45ea-a1f5-42d09d184a07","Type":"ContainerDied","Data":"07973c529ce482929bfc3ef559b47e1bf7c6b187c4e1ebb1f6ca67544a53ff9c"}
Nov 03 06:11:03.200686 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.200641    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"]
Nov 03 06:11:03.209971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.209933    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-df7236-xw7n6"]
Nov 03 06:11:03.757761 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.757723    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5" path="/var/lib/kubelet/pods/5e59ff32-d3b9-4011-a8ae-2ccdfa4294f5/volumes"
Nov 03 06:11:03.758440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.758418    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4dcb714-551f-4330-9546-d0fa4e1b8628" path="/var/lib/kubelet/pods/f4dcb714-551f-4330-9546-d0fa4e1b8628/volumes"
Nov 03 06:11:03.901854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.901814    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"]
Nov 03 06:11:03.912105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:03.912077    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-86728b-qpcv4"]
Nov 03 06:11:04.143354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.143325    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:04.205257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.205221    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\") pod \"75569012-1c4e-45ea-a1f5-42d09d184a07\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") "
Nov 03 06:11:04.205415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.205324    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\" (UniqueName: \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\") pod \"75569012-1c4e-45ea-a1f5-42d09d184a07\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") "
Nov 03 06:11:04.205415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.205379    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\") pod \"75569012-1c4e-45ea-a1f5-42d09d184a07\" (UID: \"75569012-1c4e-45ea-a1f5-42d09d184a07\") "
Nov 03 06:11:04.205843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.205770    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca" pod "75569012-1c4e-45ea-a1f5-42d09d184a07" (UID: "75569012-1c4e-45ea-a1f5-42d09d184a07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:04.206095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.206064    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:04.207704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.207659    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm" pod "75569012-1c4e-45ea-a1f5-42d09d184a07" (UID: "75569012-1c4e-45ea-a1f5-42d09d184a07"). InnerVolumeSpecName "kube-api-access-dv6sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:04.207993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.207972    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca" pod "75569012-1c4e-45ea-a1f5-42d09d184a07" (UID: "75569012-1c4e-45ea-a1f5-42d09d184a07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:04.307654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.307616    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75569012-1c4e-45ea-a1f5-42d09d184a07-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:04.307654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.307651    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv6sm\" (UniqueName: \"kubernetes.io/projected/75569012-1c4e-45ea-a1f5-42d09d184a07-kube-api-access-dv6sm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:04.940832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.940746    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"]
Nov 03 06:11:04.941418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.941402    4528 cpu_manager.go:414] "containerMap: removing container" podUID="75569012-1c4e-45ea-a1f5-42d09d184a07" containerName="revoke-keys"
Nov 03 06:11:04.941465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.941419    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75569012-1c4e-45ea-a1f5-42d09d184a07" containerName="revoke-keys"
Nov 03 06:11:04.941537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.941528    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="75569012-1c4e-45ea-a1f5-42d09d184a07" containerName="revoke-keys"
Nov 03 06:11:04.946048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.946030    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:04.985917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.985880    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"]
Nov 03 06:11:04.990484 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.990457    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"
Nov 03 06:11:04.990618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.990511    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt" event={"ID":"75569012-1c4e-45ea-a1f5-42d09d184a07","Type":"ContainerDied","Data":"0bd6449471996b2f8d1e08d8e6c4f5d8bc9f33d2be8108307dae590a017785d5"}
Nov 03 06:11:04.990618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:04.990574    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bd6449471996b2f8d1e08d8e6c4f5d8bc9f33d2be8108307dae590a017785d5"
Nov 03 06:11:05.018001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.017967    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.018172 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.018034    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.018239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.018174    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2bn\" (UniqueName: \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.119292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.119253    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.119481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.119309    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.119481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.119343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2bn\" (UniqueName: \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.119981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.119959    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.121697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.121675    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.151284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.151248    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2bn\" (UniqueName: \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\") pod \"maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.263508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.263431    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:05.574187 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:11:05.574138    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aae8124_635d_43fb_8137_42894ae3663e.slice/crio-9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae WatchSource:0}: Error finding container 9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae: Status 404 returned error can't find the container with id 9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae
Nov 03 06:11:05.575980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.575945    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"]
Nov 03 06:11:05.750027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.749997    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dba591a-cce5-4d60-a572-c5de8209e9b2" path="/var/lib/kubelet/pods/3dba591a-cce5-4d60-a572-c5de8209e9b2/volumes"
Nov 03 06:11:05.795095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:05.795062    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:11:06.005302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:06.005259    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv" event={"ID":"1aae8124-635d-43fb-8137-42894ae3663e","Type":"ContainerStarted","Data":"a88f6cf4552ec7fa4cc1d2cb466302ff3690a0abac0c5e29fd76a03743606893"}
Nov 03 06:11:06.005749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:06.005330    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv" event={"ID":"1aae8124-635d-43fb-8137-42894ae3663e","Type":"ContainerStarted","Data":"9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae"}
Nov 03 06:11:06.051621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:06.051567    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv" podStartSLOduration=2.051551923 podStartE2EDuration="2.051551923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:11:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:11:06.045519413 +0000 UTC m=+954.456916503" watchObservedRunningTime="2026-11-03 06:11:06.051551923 +0000 UTC m=+954.462949029"
Nov 03 06:11:06.292751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:06.292658    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:06.292967 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:06.292670    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:07.018126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:07.018088    4528 generic.go:356] "Generic (PLEG): container finished" podID="1aae8124-635d-43fb-8137-42894ae3663e" containerID="a88f6cf4552ec7fa4cc1d2cb466302ff3690a0abac0c5e29fd76a03743606893" exitCode=0
Nov 03 06:11:07.018609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:07.018165    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv" event={"ID":"1aae8124-635d-43fb-8137-42894ae3663e","Type":"ContainerDied","Data":"a88f6cf4552ec7fa4cc1d2cb466302ff3690a0abac0c5e29fd76a03743606893"}
Nov 03 06:11:08.206905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.206878    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:08.278265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.278185    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\") pod \"1aae8124-635d-43fb-8137-42894ae3663e\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") "
Nov 03 06:11:08.278427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.278292    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\") pod \"1aae8124-635d-43fb-8137-42894ae3663e\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") "
Nov 03 06:11:08.278532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.278493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\" (UniqueName: \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\") pod \"1aae8124-635d-43fb-8137-42894ae3663e\" (UID: \"1aae8124-635d-43fb-8137-42894ae3663e\") "
Nov 03 06:11:08.278715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.278690    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca" pod "1aae8124-635d-43fb-8137-42894ae3663e" (UID: "1aae8124-635d-43fb-8137-42894ae3663e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:08.279170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.279140    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:08.281016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.280974    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca" pod "1aae8124-635d-43fb-8137-42894ae3663e" (UID: "1aae8124-635d-43fb-8137-42894ae3663e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:08.281117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.281013    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn" pod "1aae8124-635d-43fb-8137-42894ae3663e" (UID: "1aae8124-635d-43fb-8137-42894ae3663e"). InnerVolumeSpecName "kube-api-access-mq2bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:08.380920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.380884    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mq2bn\" (UniqueName: \"kubernetes.io/projected/1aae8124-635d-43fb-8137-42894ae3663e-kube-api-access-mq2bn\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:08.380920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:08.380921    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1aae8124-635d-43fb-8137-42894ae3663e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:09.040526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:09.040495    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"
Nov 03 06:11:09.040526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:09.040519    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv" event={"ID":"1aae8124-635d-43fb-8137-42894ae3663e","Type":"ContainerDied","Data":"9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae"}
Nov 03 06:11:09.040723 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:09.040572    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f59a19738041475fe0d55ef1f7885e08ba7a8fcf6cab6a2788b5f2004545dae"
Nov 03 06:11:09.768292 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:09.768252    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:11:11.614568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.614532    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:11:11.662064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.662022    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:11:11.922679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.922560    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"]
Nov 03 06:11:11.924181 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.924138    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1aae8124-635d-43fb-8137-42894ae3663e" containerName="revoke-keys"
Nov 03 06:11:11.924181 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.924183    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aae8124-635d-43fb-8137-42894ae3663e" containerName="revoke-keys"
Nov 03 06:11:11.924426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.924411    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aae8124-635d-43fb-8137-42894ae3663e" containerName="revoke-keys"
Nov 03 06:11:11.934277 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.934248    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:11.956619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.954847    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:11:11.956619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.955666    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:11:11.956619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.956117    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:11:11.962700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.962676    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:11:11.992950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:11.992894    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"]
Nov 03 06:11:12.003917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.003882    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"]
Nov 03 06:11:12.017141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.017105    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"]
Nov 03 06:11:12.017415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.017239    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.031959 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.031924    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.032084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.031973    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.032143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.032121    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x47dh\" (UniqueName: \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133343    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x47dh\" (UniqueName: \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133508    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fh6jf\" (UniqueName: \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133535    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133563    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133597    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.134423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.133655    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.138175 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.138142    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.148101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.148070    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:11:12.155711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.155677    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.170813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.170413    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:11:12.180538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.180513    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x47dh\" (UniqueName: \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\") pod \"maas-api-revoke-sub-d6428dff-xsspb\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.242149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.238512    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fh6jf\" (UniqueName: \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.242149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.239089    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.242149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.240172    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.242149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.240334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.260461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.260390    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.266161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.266116    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fh6jf\" (UniqueName: \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\") pod \"maas-api-revoke-sub-b6e5b33b-kfznl\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.272123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.271995    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:12.308703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.308678    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:11:12.346310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.344208    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:12.414040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.413683    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:11:12.445269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.444696    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\") pod \"26702801-3755-47d7-95ca-37207e03af26\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") "
Nov 03 06:11:12.445269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.444990    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\" (UniqueName: \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\") pod \"26702801-3755-47d7-95ca-37207e03af26\" (UID: \"26702801-3755-47d7-95ca-37207e03af26\") "
Nov 03 06:11:12.445482 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.445313    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume" pod "26702801-3755-47d7-95ca-37207e03af26" (UID: "26702801-3755-47d7-95ca-37207e03af26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:12.445975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.445951    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26702801-3755-47d7-95ca-37207e03af26-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:12.449134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.449113    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm" pod "26702801-3755-47d7-95ca-37207e03af26" (UID: "26702801-3755-47d7-95ca-37207e03af26"). InnerVolumeSpecName "kube-api-access-4cmfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:12.581816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.576292    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cmfm\" (UniqueName: \"kubernetes.io/projected/26702801-3755-47d7-95ca-37207e03af26-kube-api-access-4cmfm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:12.650124 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.650089    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"]
Nov 03 06:11:12.652448 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:11:12.652419    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244f3d91_8ec4_4b62_bc20_585dd79e9809.slice/crio-9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127 WatchSource:0}: Error finding container 9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127: Status 404 returned error can't find the container with id 9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127
Nov 03 06:11:12.677239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.677209    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\") pod \"df8416e0-a375-4239-a542-f8c84c98e548\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") "
Nov 03 06:11:12.677397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.677370    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\" (UniqueName: \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\") pod \"df8416e0-a375-4239-a542-f8c84c98e548\" (UID: \"df8416e0-a375-4239-a542-f8c84c98e548\") "
Nov 03 06:11:12.677551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.677532    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume" pod "df8416e0-a375-4239-a542-f8c84c98e548" (UID: "df8416e0-a375-4239-a542-f8c84c98e548"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:12.678266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.678237    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df8416e0-a375-4239-a542-f8c84c98e548-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:12.680086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.680062    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr" pod "df8416e0-a375-4239-a542-f8c84c98e548" (UID: "df8416e0-a375-4239-a542-f8c84c98e548"). InnerVolumeSpecName "kube-api-access-8bdcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:12.699552 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:11:12.696341    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb76eb_d1c9_4b47_afca_ad805468b5ca.slice/crio-a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5 WatchSource:0}: Error finding container a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5: Status 404 returned error can't find the container with id a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5
Nov 03 06:11:12.699552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.699074    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"]
Nov 03 06:11:12.779998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:12.779961    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdcr\" (UniqueName: \"kubernetes.io/projected/df8416e0-a375-4239-a542-f8c84c98e548-kube-api-access-8bdcr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:13.031006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.030963    4528 scope.go:122] "RemoveContainer" containerID="91125266ea9ffc17ff9ecee8b2ea9cba70b121644bbf1eaba7932fe58b4c0f40"
Nov 03 06:11:13.083593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.083563    4528 scope.go:122] "RemoveContainer" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"
Nov 03 06:11:13.090137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.090111    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"
Nov 03 06:11:13.090290 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.090103    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" event={"ID":"df8416e0-a375-4239-a542-f8c84c98e548","Type":"ContainerDied","Data":"91125266ea9ffc17ff9ecee8b2ea9cba70b121644bbf1eaba7932fe58b4c0f40"}
Nov 03 06:11:13.090290 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.090268    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t" event={"ID":"df8416e0-a375-4239-a542-f8c84c98e548","Type":"ContainerDied","Data":"0fa696c0e2eb3468b5ccce4ace376493b659c6f51a1b4444173e8fdc9361ae36"}
Nov 03 06:11:13.091910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.091885    4528 generic.go:356] "Generic (PLEG): container finished" podID="26702801-3755-47d7-95ca-37207e03af26" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c" exitCode=0
Nov 03 06:11:13.092046 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.091925    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"
Nov 03 06:11:13.092046 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.091958    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" event={"ID":"26702801-3755-47d7-95ca-37207e03af26","Type":"ContainerDied","Data":"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"}
Nov 03 06:11:13.092046 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.092002    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw" event={"ID":"26702801-3755-47d7-95ca-37207e03af26","Type":"ContainerDied","Data":"c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37"}
Nov 03 06:11:13.092208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.092031    4528 scope.go:122] "RemoveContainer" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"
Nov 03 06:11:13.101597 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:13.101252    4528 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw_openshift-ingress_26702801-3755-47d7-95ca-37207e03af26_0 in pod sandbox c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37 from index: no such id: 'c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c'" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"
Nov 03 06:11:13.101597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.101300    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"} err="rpc error: code = Unknown desc = failed to delete container k8s_payload-processing_payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw_openshift-ingress_26702801-3755-47d7-95ca-37207e03af26_0 in pod sandbox c68a4568ec3c7a292cefef95be7fde98dd808cbd9fec40b8253f4737297c6e37 from index: no such id: 'c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c'"
Nov 03 06:11:13.101597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.101329    4528 scope.go:122] "RemoveContainer" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"
Nov 03 06:11:13.104517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.102082    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl" event={"ID":"23cb76eb-d1c9-4b47-afca-ad805468b5ca","Type":"ContainerStarted","Data":"a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5"}
Nov 03 06:11:13.104517 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:13.102162    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c\": container with ID starting with c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c not found: ID does not exist" containerID="c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"
Nov 03 06:11:13.104517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.102190    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c"} err="failed to get container status \"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c\": rpc error: code = NotFound desc = could not find container \"c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c\": container with ID starting with c75a44a943ca07967bdd4170b551c50850d483df43edadc55bb072c37f9c6c9c not found: ID does not exist"
Nov 03 06:11:13.109070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.109045    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb" event={"ID":"244f3d91-8ec4-4b62-bc20-585dd79e9809","Type":"ContainerStarted","Data":"628f03adb0f2a660cbceecaf1015fdc096529d1d9f3fb9e1fdf9c21d8314f7be"}
Nov 03 06:11:13.109174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.109102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb" event={"ID":"244f3d91-8ec4-4b62-bc20-585dd79e9809","Type":"ContainerStarted","Data":"9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127"}
Nov 03 06:11:13.174310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.174263    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb" podStartSLOduration=2.174248278 podStartE2EDuration="2.174248278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:11:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:11:13.1687379 +0000 UTC m=+961.580134978" watchObservedRunningTime="2026-11-03 06:11:13.174248278 +0000 UTC m=+961.585645360"
Nov 03 06:11:13.233978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.233946    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:11:13.243272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.243246    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-df7236-68858c7cc9-jsmtw"]
Nov 03 06:11:13.274675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.274633    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:11:13.282382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.282354    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-df7236-6cf8948b79-22v4t"]
Nov 03 06:11:13.694491 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:13.694449    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:11:13.749973 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.749936    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26702801-3755-47d7-95ca-37207e03af26" path="/var/lib/kubelet/pods/26702801-3755-47d7-95ca-37207e03af26/volumes"
Nov 03 06:11:13.750404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:13.750385    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8416e0-a375-4239-a542-f8c84c98e548" path="/var/lib/kubelet/pods/df8416e0-a375-4239-a542-f8c84c98e548/volumes"
Nov 03 06:11:14.137876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:14.137837    4528 generic.go:356] "Generic (PLEG): container finished" podID="23cb76eb-d1c9-4b47-afca-ad805468b5ca" containerID="f4a486325fcc72cb4fc552535d1eb8ccdb2c91fda1a2369abc9cec7bd616bd7c" exitCode=0
Nov 03 06:11:14.138042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:14.137932    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl" event={"ID":"23cb76eb-d1c9-4b47-afca-ad805468b5ca","Type":"ContainerDied","Data":"f4a486325fcc72cb4fc552535d1eb8ccdb2c91fda1a2369abc9cec7bd616bd7c"}
Nov 03 06:11:14.139779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:14.139756    4528 generic.go:356] "Generic (PLEG): container finished" podID="244f3d91-8ec4-4b62-bc20-585dd79e9809" containerID="628f03adb0f2a660cbceecaf1015fdc096529d1d9f3fb9e1fdf9c21d8314f7be" exitCode=0
Nov 03 06:11:14.139930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:14.139819    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb" event={"ID":"244f3d91-8ec4-4b62-bc20-585dd79e9809","Type":"ContainerDied","Data":"628f03adb0f2a660cbceecaf1015fdc096529d1d9f3fb9e1fdf9c21d8314f7be"}
Nov 03 06:11:15.110712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.110670    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:11:15.111212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.110907    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy" containerID="cri-o://1e2721d0ba3e55a5b29a206b883a1bbbfe09bd2d5ad63e74af14ae8797a1f0fe" gracePeriod=30
Nov 03 06:11:15.122042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.122006    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:11:15.122225 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.122203    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" containerID="cri-o://7e2ac5c2bcca9e6dd17f650486c3c79e820def2d7d3dedcd1d2a413004259cc8" gracePeriod=30
Nov 03 06:11:15.170244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.170206    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:11:15.170554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.170525    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" containerID="cri-o://b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674" gracePeriod=30
Nov 03 06:11:15.208513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.208440    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:11:15.211851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.208747    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" containerID="cri-o://c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026" gracePeriod=30
Nov 03 06:11:15.499619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.499592    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:15.503903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.503876    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:15.558530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.558496    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\" (UniqueName: \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\") pod \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") "
Nov 03 06:11:15.558709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.558604    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\") pod \"244f3d91-8ec4-4b62-bc20-585dd79e9809\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") "
Nov 03 06:11:15.558803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.558736    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\") pod \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") "
Nov 03 06:11:15.558803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.558779    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\" (UniqueName: \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\") pod \"244f3d91-8ec4-4b62-bc20-585dd79e9809\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") "
Nov 03 06:11:15.559194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.558966    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\") pod \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\" (UID: \"23cb76eb-d1c9-4b47-afca-ad805468b5ca\") "
Nov 03 06:11:15.559194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.559020    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\") pod \"244f3d91-8ec4-4b62-bc20-585dd79e9809\" (UID: \"244f3d91-8ec4-4b62-bc20-585dd79e9809\") "
Nov 03 06:11:15.559194 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.559123    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca" pod "23cb76eb-d1c9-4b47-afca-ad805468b5ca" (UID: "23cb76eb-d1c9-4b47-afca-ad805468b5ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:15.559553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.559531    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.559619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.559570    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca" pod "244f3d91-8ec4-4b62-bc20-585dd79e9809" (UID: "244f3d91-8ec4-4b62-bc20-585dd79e9809"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:15.561227 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.561204    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca" pod "23cb76eb-d1c9-4b47-afca-ad805468b5ca" (UID: "23cb76eb-d1c9-4b47-afca-ad805468b5ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:15.561942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.561918    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca" pod "244f3d91-8ec4-4b62-bc20-585dd79e9809" (UID: "244f3d91-8ec4-4b62-bc20-585dd79e9809"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:15.561942 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.561927    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf" pod "23cb76eb-d1c9-4b47-afca-ad805468b5ca" (UID: "23cb76eb-d1c9-4b47-afca-ad805468b5ca"). InnerVolumeSpecName "kube-api-access-fh6jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:15.563204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.563180    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh" pod "244f3d91-8ec4-4b62-bc20-585dd79e9809" (UID: "244f3d91-8ec4-4b62-bc20-585dd79e9809"). InnerVolumeSpecName "kube-api-access-x47dh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:15.648412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.648378    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:11:15.661278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.661250    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x47dh\" (UniqueName: \"kubernetes.io/projected/244f3d91-8ec4-4b62-bc20-585dd79e9809-kube-api-access-x47dh\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.661391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.661280    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23cb76eb-d1c9-4b47-afca-ad805468b5ca-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.661391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.661296    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.661391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.661307    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fh6jf\" (UniqueName: \"kubernetes.io/projected/23cb76eb-d1c9-4b47-afca-ad805468b5ca-kube-api-access-fh6jf\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.661391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:15.661316    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/244f3d91-8ec4-4b62-bc20-585dd79e9809-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:15.831560 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:15.831522    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23cb76eb_d1c9_4b47_afca_ad805468b5ca.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod244f3d91_8ec4_4b62_bc20_585dd79e9809.slice\": RecentStats: unable to find data in memory cache]"
Nov 03 06:11:16.152221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.152176    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:11:16.152683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.152429    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" containerID="cri-o://a265c11abb5ae06013da3d718b875c6a20d54bddc711961b3212e81469735e66" gracePeriod=30
Nov 03 06:11:16.162515 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.162491    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"
Nov 03 06:11:16.162656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.162510    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl" event={"ID":"23cb76eb-d1c9-4b47-afca-ad805468b5ca","Type":"ContainerDied","Data":"a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5"}
Nov 03 06:11:16.162656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.162565    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a70fe9357e159a5977edd218c111b1d75667c87707a32e72b76fbeb9528429e5"
Nov 03 06:11:16.164051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.164018    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb" event={"ID":"244f3d91-8ec4-4b62-bc20-585dd79e9809","Type":"ContainerDied","Data":"9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127"}
Nov 03 06:11:16.164137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.164075    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c6ff84b9517dc140bf5a303462e5145bbf767b8c3aab7f8503c46c2a18b8127"
Nov 03 06:11:16.164137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.164038    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"
Nov 03 06:11:16.180979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.180952    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:11:16.181644 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.181172    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" containerID="cri-o://6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3" gracePeriod=30
Nov 03 06:11:16.188685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.188660    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:11:16.188859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.188838    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" containerID="cri-o://8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6" gracePeriod=30
Nov 03 06:11:16.532874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.524989    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:16.532874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.525103    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:16.615758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.615711    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing" containerID="cri-o://c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0" gracePeriod=25
Nov 03 06:11:16.662773 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.662710    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing" containerID="cri-o://58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61" gracePeriod=25
Nov 03 06:11:16.759313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.759270    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:11:16.759525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.759500    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" containerID="cri-o://68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe" gracePeriod=30
Nov 03 06:11:16.778278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.778239    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:11:16.778557 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.778530    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy" containerID="cri-o://339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470" gracePeriod=30
Nov 03 06:11:16.788665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.788607    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:11:16.788898 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:16.788783    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" containerID="cri-o://aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3" gracePeriod=30
Nov 03 06:11:17.084028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.083989    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:17.084214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.084047    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:17.299430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.299394    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:17.299938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.299402    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:17.811067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.811022    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:17.811283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.811088    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:17.879817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.872967    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:17.879817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:17.873028    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:18.009816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:18.002274    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:18.009816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:18.002342    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:19.170753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:19.170705    4528 patch_prober.go:29] interesting pod/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:19.171202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:19.170781    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:19.235191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:19.235151    4528 patch_prober.go:29] interesting pod/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:19.235350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:19.235220    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:20.085582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.085545    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"]
Nov 03 06:11:20.086168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086154    4528 cpu_manager.go:414] "containerMap: removing container" podUID="244f3d91-8ec4-4b62-bc20-585dd79e9809" containerName="revoke-keys"
Nov 03 06:11:20.086215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086169    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="244f3d91-8ec4-4b62-bc20-585dd79e9809" containerName="revoke-keys"
Nov 03 06:11:20.086215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086183    4528 cpu_manager.go:414] "containerMap: removing container" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing"
Nov 03 06:11:20.086215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086190    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing"
Nov 03 06:11:20.086215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086204    4528 cpu_manager.go:414] "containerMap: removing container" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing"
Nov 03 06:11:20.086215 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086209    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086221    4528 cpu_manager.go:414] "containerMap: removing container" podUID="23cb76eb-d1c9-4b47-afca-ad805468b5ca" containerName="revoke-keys"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086227    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23cb76eb-d1c9-4b47-afca-ad805468b5ca" containerName="revoke-keys"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086329    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="26702801-3755-47d7-95ca-37207e03af26" containerName="payload-processing"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086338    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="23cb76eb-d1c9-4b47-afca-ad805468b5ca" containerName="revoke-keys"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086351    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="244f3d91-8ec4-4b62-bc20-585dd79e9809" containerName="revoke-keys"
Nov 03 06:11:20.086385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.086357    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8416e0-a375-4239-a542-f8c84c98e548" containerName="payload-pre-processing"
Nov 03 06:11:20.091496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.091475    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.112519 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.112489    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"]
Nov 03 06:11:20.117219 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.117197    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:11:20.117753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.117734    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.123895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.123871    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:11:20.124027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.123999    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:11:20.124087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.124065    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:11:20.145460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.145429    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"]
Nov 03 06:11:20.155179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.155155    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"]
Nov 03 06:11:20.185495 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185454    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfbc\" (UniqueName: \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.185865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185600    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zw96\" (UniqueName: \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.185865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185649    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.185865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185726    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.185865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185806    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.186032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.185880    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.272110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.272081    4528 generic.go:356] "Generic (PLEG): container finished" podID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerID="1e2721d0ba3e55a5b29a206b883a1bbbfe09bd2d5ad63e74af14ae8797a1f0fe" exitCode=0
Nov 03 06:11:20.272251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.272148    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerDied","Data":"1e2721d0ba3e55a5b29a206b883a1bbbfe09bd2d5ad63e74af14ae8797a1f0fe"}
Nov 03 06:11:20.272251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.272217    4528 scope.go:122] "RemoveContainer" containerID="268034fc00ad8510d12d206023f2c4416dcb4ebd8b180a647f30de82e331b1a9"
Nov 03 06:11:20.274045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.274024    4528 generic.go:356] "Generic (PLEG): container finished" podID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerID="7e2ac5c2bcca9e6dd17f650486c3c79e820def2d7d3dedcd1d2a413004259cc8" exitCode=0
Nov 03 06:11:20.274150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.274058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" event={"ID":"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57","Type":"ContainerDied","Data":"7e2ac5c2bcca9e6dd17f650486c3c79e820def2d7d3dedcd1d2a413004259cc8"}
Nov 03 06:11:20.287371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.287340    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zw96\" (UniqueName: \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.287510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.287471    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.288108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.288083    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.288197 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.288136    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.288264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.288246    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.288405 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.288389    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfbc\" (UniqueName: \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.288727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.288710    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.289223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.289202    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.291066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.291041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.291157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.291048    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.334336 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.334264    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfbc\" (UniqueName: \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\") pod \"maas-api-revoke-sub-227fa5ae-p57gq\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.337091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.337026    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zw96\" (UniqueName: \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\") pod \"maas-api-revoke-sub-cdc8d99f-n47fl\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.414510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.408039    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:20.448230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.448196    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:20.615825 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.615780    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:11:20.618156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.618127    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:11:20.649260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.648894    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing" containerID="cri-o://4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2" gracePeriod=25
Nov 03 06:11:20.708593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.708564    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:11:20.731549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.731506    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:11:20.803653 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.803617    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.803970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.803930    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.804257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.804238    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.805122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.805102    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.805293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.805278    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.806152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.806131    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.806914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.806764    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.807083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.807063    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.807844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.807826    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.808547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.808261    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.809023 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.808687    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.809456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.809172    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.809618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.809600    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.809747 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.809731    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.809891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.809874    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.810029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.810012    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.810142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.810125    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.810347 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.810330    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.810931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.810912    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.811088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.811070    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.811214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.811197    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.811394 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.811376    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.811545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.811528    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.811688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.811673    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.813260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813240    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.813425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813407    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.813564 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813547    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.813721 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813702    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.813851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\") pod \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\" (UID: \"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57\") "
Nov 03 06:11:20.813982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.813965    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.814103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.814085    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\") pod \"3144501e-3a21-4303-9bbc-afad4fc6dafa\" (UID: \"3144501e-3a21-4303-9bbc-afad4fc6dafa\") "
Nov 03 06:11:20.815072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.815052    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.815205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.815190    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.815819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.808196    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.815940 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.808992    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.816019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.815781    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.830171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.829472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.834285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.833543    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.835067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.835039    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.837847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.837823    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.840120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.840096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:20.840487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.840441    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.840831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.840784    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.841144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.841126    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.843606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.843467    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.845379 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.845018    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main" containerID="cri-o://fce25e2c7f6f170f3f216a9471c7a3b99f77d9b5856725b47852efb5087c270c" gracePeriod=45
Nov 03 06:11:20.846022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.845613    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:20.846022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.845852    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.846263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.846166    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.846328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.846255    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.846404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.846379    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.846925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.846634    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:20.851165 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.851097    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:20.852933 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.852885    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx" pod "3144501e-3a21-4303-9bbc-afad4fc6dafa" (UID: "3144501e-3a21-4303-9bbc-afad4fc6dafa"). InnerVolumeSpecName "kube-api-access-l7jkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.853027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.852944    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.853255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.853221    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "kube-api-access-xtfjw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.855425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.855397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.860206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.859385    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.865876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.865849    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"]
Nov 03 06:11:20.868224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.867860    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "kube-api-access-b79wv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.876251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.876211    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:20.888140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.888113    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token" pod "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" (UID: "5c03e07e-1e7a-4ab6-a890-0a379fc2ad57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.912712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.912680    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"]
Nov 03 06:11:20.913358 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:11:20.913332    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb3b4a50_0498_4d01_894c_591c037e18f2.slice/crio-163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8 WatchSource:0}: Error finding container 163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8: Status 404 returned error can't find the container with id 163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8
Nov 03 06:11:20.916748 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.916724    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.916881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.916777    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.916881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.916840    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.916881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.916874    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.917057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917035    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.917111 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917096    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\") pod \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\" (UID: \"e8b9b9c0-cf19-439f-aaad-6b654d2b2768\") "
Nov 03 06:11:20.917143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917117    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:20.917143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917137    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\") pod \"ea3d0c16-444e-495c-8208-b505170d2aef\" (UID: \"ea3d0c16-444e-495c-8208-b505170d2aef\") "
Nov 03 06:11:20.917833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917811    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.917833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917832    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ea3d0c16-444e-495c-8208-b505170d2aef-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917841    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917850    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917858    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917866    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917873    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917881    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b79wv\" (UniqueName: \"kubernetes.io/projected/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-kube-api-access-b79wv\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917891    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917899    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917906    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917913    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917920    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917927    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ea3d0c16-444e-495c-8208-b505170d2aef-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917936    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917944    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917952    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917961    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917970    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917977    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xtfjw\" (UniqueName: \"kubernetes.io/projected/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-kube-api-access-xtfjw\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917986    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7jkx\" (UniqueName: \"kubernetes.io/projected/3144501e-3a21-4303-9bbc-afad4fc6dafa-kube-api-access-l7jkx\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.917994    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3144501e-3a21-4303-9bbc-afad4fc6dafa-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918002    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3144501e-3a21-4303-9bbc-afad4fc6dafa-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.918000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918010    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3144501e-3a21-4303-9bbc-afad4fc6dafa-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918018    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918028    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918036    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918044    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918323    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918538    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918588    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.919174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.918721    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:20.919543 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.919486    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9" pod "ea3d0c16-444e-495c-8208-b505170d2aef" (UID: "ea3d0c16-444e-495c-8208-b505170d2aef"). InnerVolumeSpecName "kube-api-access-4crt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:20.921362 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:20.921337    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo" pod "e8b9b9c0-cf19-439f-aaad-6b654d2b2768" (UID: "e8b9b9c0-cf19-439f-aaad-6b654d2b2768"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:21.018961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018931    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4crt9\" (UniqueName: \"kubernetes.io/projected/ea3d0c16-444e-495c-8208-b505170d2aef-kube-api-access-4crt9\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.018961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018957    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.018961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018967    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ea3d0c16-444e-495c-8208-b505170d2aef-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.019158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018977    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.019158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018986    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.019158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.018994    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8b9b9c0-cf19-439f-aaad-6b654d2b2768-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.285742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.285715    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"
Nov 03 06:11:21.285742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.285721    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj" event={"ID":"3144501e-3a21-4303-9bbc-afad4fc6dafa","Type":"ContainerDied","Data":"19216113ab35b167fb3ff4840763c3830c7e0e320500a27d2c4e82d56936d248"}
Nov 03 06:11:21.286307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.285855    4528 scope.go:122] "RemoveContainer" containerID="1e2721d0ba3e55a5b29a206b883a1bbbfe09bd2d5ad63e74af14ae8797a1f0fe"
Nov 03 06:11:21.287998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.287943    4528 generic.go:356] "Generic (PLEG): container finished" podID="95980fca-5c88-40d0-9004-1609238d9708" containerID="a265c11abb5ae06013da3d718b875c6a20d54bddc711961b3212e81469735e66" exitCode=0
Nov 03 06:11:21.288064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.287985    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" event={"ID":"95980fca-5c88-40d0-9004-1609238d9708","Type":"ContainerDied","Data":"a265c11abb5ae06013da3d718b875c6a20d54bddc711961b3212e81469735e66"}
Nov 03 06:11:21.290156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.290136    4528 generic.go:356] "Generic (PLEG): container finished" podID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerID="b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674" exitCode=0
Nov 03 06:11:21.290257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.290213    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"
Nov 03 06:11:21.290257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.290215    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" event={"ID":"e8b9b9c0-cf19-439f-aaad-6b654d2b2768","Type":"ContainerDied","Data":"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"}
Nov 03 06:11:21.290382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.290264    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w" event={"ID":"e8b9b9c0-cf19-439f-aaad-6b654d2b2768","Type":"ContainerDied","Data":"464dd19cf2b249fe75f0153b5fe5b2f88786872273e93d44deea28374a5d71a7"}
Nov 03 06:11:21.291719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.291678    4528 generic.go:356] "Generic (PLEG): container finished" podID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerID="fce25e2c7f6f170f3f216a9471c7a3b99f77d9b5856725b47852efb5087c270c" exitCode=0
Nov 03 06:11:21.291821 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.291762    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" event={"ID":"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67","Type":"ContainerDied","Data":"fce25e2c7f6f170f3f216a9471c7a3b99f77d9b5856725b47852efb5087c270c"}
Nov 03 06:11:21.291885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.291823    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x" event={"ID":"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67","Type":"ContainerDied","Data":"489c7679dedb3aedc21d97423ef70a5577f47545a57c994d800352a81d758eb0"}
Nov 03 06:11:21.291885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.291842    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="489c7679dedb3aedc21d97423ef70a5577f47545a57c994d800352a81d758eb0"
Nov 03 06:11:21.295234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.295213    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"
Nov 03 06:11:21.295342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.295236    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl" event={"ID":"5c03e07e-1e7a-4ab6-a890-0a379fc2ad57","Type":"ContainerDied","Data":"9e1bb24ddb277ebaf9f4d41d08d52a6aaefb46f5389a0409dc29e08e59a85ce1"}
Nov 03 06:11:21.297119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.297098    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl" event={"ID":"fb3b4a50-0498-4d01-894c-591c037e18f2","Type":"ContainerStarted","Data":"163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8"}
Nov 03 06:11:21.300573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.300446    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq" event={"ID":"6cb6d8e8-20ff-44f0-a158-592cc43461ff","Type":"ContainerStarted","Data":"4855e4dfe832c2bd66408487f7d54c2562cd60884e596706bc216c65198f78e5"}
Nov 03 06:11:21.342017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.336582    4528 generic.go:356] "Generic (PLEG): container finished" podID="ea3d0c16-444e-495c-8208-b505170d2aef" containerID="c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026" exitCode=0
Nov 03 06:11:21.342017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.336667    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" event={"ID":"ea3d0c16-444e-495c-8208-b505170d2aef","Type":"ContainerDied","Data":"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"}
Nov 03 06:11:21.342017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.336720    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p" event={"ID":"ea3d0c16-444e-495c-8208-b505170d2aef","Type":"ContainerDied","Data":"c707b8d6fbe683d9fb7ddb949c43e390a24fb5aeac9ff0ad3678019a0179d8eb"}
Nov 03 06:11:21.342017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.336847    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"
Nov 03 06:11:21.392670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.392641    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:11:21.393597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.393577    4528 scope.go:122] "RemoveContainer" containerID="b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"
Nov 03 06:11:21.435614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435583    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435652    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\" (UniqueName: \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435753    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435811    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\" (UniqueName: \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435915    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\") pod \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\" (UID: \"45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67\") "
Nov 03 06:11:21.435945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.435915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.436369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.436144    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.436465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.436442    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.436538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.436464    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.436932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.436913    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.441576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.441550    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:21.441667 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.441550    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.441716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.441638    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m" pod "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" (UID: "45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67"). InnerVolumeSpecName "kube-api-access-g4m6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.457768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.457737    4528 scope.go:122] "RemoveContainer" containerID="b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"
Nov 03 06:11:21.476913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.465627    4528 patch_prober.go:29] interesting pod/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:21.476913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.465687    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:21.476913 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:21.468600    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674\": container with ID starting with b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674 not found: ID does not exist" containerID="b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"
Nov 03 06:11:21.476913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.468644    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674"} err="failed to get container status \"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674\": rpc error: code = NotFound desc = could not find container \"b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674\": container with ID starting with b5764c37a1a8a80f72b8c8966c6775bc3542de63bbe7bd7b6beed662bff69674 not found: ID does not exist"
Nov 03 06:11:21.476913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.468668    4528 scope.go:122] "RemoveContainer" containerID="7e2ac5c2bcca9e6dd17f650486c3c79e820def2d7d3dedcd1d2a413004259cc8"
Nov 03 06:11:21.493984 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.493954    4528 scope.go:122] "RemoveContainer" containerID="c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"
Nov 03 06:11:21.523029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.523001    4528 scope.go:122] "RemoveContainer" containerID="c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"
Nov 03 06:11:21.527632 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:21.527535    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026\": container with ID starting with c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026 not found: ID does not exist" containerID="c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"
Nov 03 06:11:21.527941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.527580    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026"} err="failed to get container status \"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026\": rpc error: code = NotFound desc = could not find container \"c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026\": container with ID starting with c8b4138f943bbed5bb91fbfd86d80286d8500d9796ce43a581a144ef1e53d026 not found: ID does not exist"
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.536740    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.536854    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.537939    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4m6m\" (UniqueName: \"kubernetes.io/projected/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-kube-api-access-g4m6m\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.537964    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.537979    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.538073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.538019    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.545399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.543279    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-pftrl"]
Nov 03 06:11:21.559196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.559170    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:11:21.587114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.587081    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:11:21.595142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.595116    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-bv86p"]
Nov 03 06:11:21.607298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.607272    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:11:21.632466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.632435    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:11:21.639119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639086    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.639257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639141    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.639257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639193    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-f8k4w"]
Nov 03 06:11:21.639257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639199    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.639430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639286    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.639430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639319    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.639430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639369    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.639538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639422    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.639538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639428    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.639538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639477    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.639675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639509    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.639675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639577    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.639675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639592    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.639675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639629    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.639675 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639667    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639708    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639710    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639752    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639787    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639855    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639903    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.639979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640012    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640072    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640103    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640196    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\") pod \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\" (UID: \"c220ecdf-1400-4b8b-8df1-ff68f00d8748\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640274    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640313    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640332    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\") pod \"95980fca-5c88-40d0-9004-1609238d9708\" (UID: \"95980fca-5c88-40d0-9004-1609238d9708\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640377    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.640525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.640408    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\") pod \"ee16f136-f2d3-41e7-9075-dff10ab7f195\" (UID: \"ee16f136-f2d3-41e7-9075-dff10ab7f195\") "
Nov 03 06:11:21.641157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641096    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.641157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641125    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.641157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641140    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.641157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641154    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/95980fca-5c88-40d0-9004-1609238d9708-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.641480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641168    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.641558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641484    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:21.641864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.641840    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.642157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642126    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.642281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642257    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "kube-api-access-2xv5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.642733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642508    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:21.642733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642583    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:21.642911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642729    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.642911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642739    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.642911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642839    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:21.643143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.642978    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.643143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.643023    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.643143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.643059    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.643302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.643246    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.643702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.643424    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.643702 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.643433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.647511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.647433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "kube-api-access-rk8v4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.647511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.647486    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.647755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.647727    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.649698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.649660    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks" pod "95980fca-5c88-40d0-9004-1609238d9708" (UID: "95980fca-5c88-40d0-9004-1609238d9708"). InnerVolumeSpecName "kube-api-access-vhxks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.650247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.650175    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:21.650462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.650430    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token" pod "ee16f136-f2d3-41e7-9075-dff10ab7f195" (UID: "ee16f136-f2d3-41e7-9075-dff10ab7f195"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:21.651463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.651442    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy" pod "c220ecdf-1400-4b8b-8df1-ff68f00d8748" (UID: "c220ecdf-1400-4b8b-8df1-ff68f00d8748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:21.676626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.676590    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:11:21.689017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.688944    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-86728b-openshift-default-6c6f76ddd7-xp8lj"]
Nov 03 06:11:21.743614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.743493    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.743614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.743528    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee16f136-f2d3-41e7-9075-dff10ab7f195-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744106 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.743542    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/95980fca-5c88-40d0-9004-1609238d9708-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744106 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744062    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744079    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744361    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744375    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744390    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xv5b\" (UniqueName: \"kubernetes.io/projected/c220ecdf-1400-4b8b-8df1-ff68f00d8748-kube-api-access-2xv5b\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744406    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744430    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhxks\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-kube-api-access-vhxks\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744444    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744459    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744472    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/95980fca-5c88-40d0-9004-1609238d9708-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744484    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744500    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744514    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744525    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8v4\" (UniqueName: \"kubernetes.io/projected/ee16f136-f2d3-41e7-9075-dff10ab7f195-kube-api-access-rk8v4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744538    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c220ecdf-1400-4b8b-8df1-ff68f00d8748-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744551    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744563    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c220ecdf-1400-4b8b-8df1-ff68f00d8748-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744577    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee16f136-f2d3-41e7-9075-dff10ab7f195-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.744611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.744589    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/95980fca-5c88-40d0-9004-1609238d9708-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:21.753853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.753827    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" path="/var/lib/kubelet/pods/3144501e-3a21-4303-9bbc-afad4fc6dafa/volumes"
Nov 03 06:11:21.754343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.754315    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" path="/var/lib/kubelet/pods/5c03e07e-1e7a-4ab6-a890-0a379fc2ad57/volumes"
Nov 03 06:11:21.754865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.754845    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" path="/var/lib/kubelet/pods/e8b9b9c0-cf19-439f-aaad-6b654d2b2768/volumes"
Nov 03 06:11:21.755626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:21.755604    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" path="/var/lib/kubelet/pods/ea3d0c16-444e-495c-8208-b505170d2aef/volumes"
Nov 03 06:11:22.055467 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.055438    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:11:22.102661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.102632    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:11:22.137934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.137909    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:11:22.150387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150324    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150406    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150450    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.150534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150492    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150526    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150568    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150624    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150655    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150691    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150769    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.150867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150847    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.150944    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151031    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151074    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151145    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\") pod \"81219190-ee1a-4ecf-acdf-c29942df3e8c\" (UID: \"81219190-ee1a-4ecf-acdf-c29942df3e8c\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151176    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.151270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151237    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\") pod \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\" (UID: \"9c5589a2-bb5c-4eb0-9931-b383d8f37850\") "
Nov 03 06:11:22.153648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151029    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.153648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.151676    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:22.153648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.152227    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.153648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.153397    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.153946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.153817    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.154156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.154002    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.155400 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.155371    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.156148 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.155940    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.156319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.156269    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.156562 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.156491    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:22.156863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.156822    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.160838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.160759    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.166087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.165931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "kube-api-access-85wmt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.166087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.166007    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:22.166087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.166062    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:22.170425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.170391    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token" pod "9c5589a2-bb5c-4eb0-9931-b383d8f37850" (UID: "9c5589a2-bb5c-4eb0-9931-b383d8f37850"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.170597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.170552    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.170965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.170931    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx" pod "81219190-ee1a-4ecf-acdf-c29942df3e8c" (UID: "81219190-ee1a-4ecf-acdf-c29942df3e8c"). InnerVolumeSpecName "kube-api-access-b5zgx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.253228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253173    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253267    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253301    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253339    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253394    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253453    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253494    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253530    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.253769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253597    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.253769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.253636    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\") pod \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\" (UID: \"8ac60968-6d3b-441f-90ad-04d6051d5f0f\") "
Nov 03 06:11:22.254214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254174    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.254338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254317    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254346    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254360    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254377    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254375    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254391    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254404    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254417    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254430    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81219190-ee1a-4ecf-acdf-c29942df3e8c-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254443    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85wmt\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-kube-api-access-85wmt\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254458    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254471    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254484    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254498    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zgx\" (UniqueName: \"kubernetes.io/projected/81219190-ee1a-4ecf-acdf-c29942df3e8c-kube-api-access-b5zgx\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254521    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254534    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254553    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81219190-ee1a-4ecf-acdf-c29942df3e8c-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254566    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254580    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c5589a2-bb5c-4eb0-9931-b383d8f37850-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254592    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81219190-ee1a-4ecf-acdf-c29942df3e8c-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254605    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.254856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254699    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.255321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.254884    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:22.256056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.256032    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:22.256886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.256860    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:22.257390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.257365    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "kube-api-access-97tvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.257497 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.257400    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token" pod "8ac60968-6d3b-441f-90ad-04d6051d5f0f" (UID: "8ac60968-6d3b-441f-90ad-04d6051d5f0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:22.353804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.353753    4528 generic.go:356] "Generic (PLEG): container finished" podID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerID="68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe" exitCode=0
Nov 03 06:11:22.354282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.353860    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"
Nov 03 06:11:22.354282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.353856    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" event={"ID":"9c5589a2-bb5c-4eb0-9931-b383d8f37850","Type":"ContainerDied","Data":"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"}
Nov 03 06:11:22.354282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.354058    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c" event={"ID":"9c5589a2-bb5c-4eb0-9931-b383d8f37850","Type":"ContainerDied","Data":"df3d13686c483f676bd6f29097094c2c1be7afa7a01c797d18016a3aeb89fd65"}
Nov 03 06:11:22.354282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.354094    4528 scope.go:122] "RemoveContainer" containerID="68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355562    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355596    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97tvq\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-kube-api-access-97tvq\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355615    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355628    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355640    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355656    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355660    4528 generic.go:356] "Generic (PLEG): container finished" podID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerID="6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3" exitCode=0
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355744    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355876    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" event={"ID":"c220ecdf-1400-4b8b-8df1-ff68f00d8748","Type":"ContainerDied","Data":"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"}
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355927    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx" event={"ID":"c220ecdf-1400-4b8b-8df1-ff68f00d8748","Type":"ContainerDied","Data":"8f046790ab0bcda00b088a7671eafc9d7beb204f920e1771852857c531d16f0a"}
Nov 03 06:11:22.356567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.355667    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ac60968-6d3b-441f-90ad-04d6051d5f0f-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:22.359516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.359477    4528 generic.go:356] "Generic (PLEG): container finished" podID="fb3b4a50-0498-4d01-894c-591c037e18f2" containerID="dc5b92a7b28b8b6bd7b976df557a0867331d4a7682e4968d9c7d73269d1659ab" exitCode=0
Nov 03 06:11:22.359627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.359557    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl" event={"ID":"fb3b4a50-0498-4d01-894c-591c037e18f2","Type":"ContainerDied","Data":"dc5b92a7b28b8b6bd7b976df557a0867331d4a7682e4968d9c7d73269d1659ab"}
Nov 03 06:11:22.362678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.362649    4528 generic.go:356] "Generic (PLEG): container finished" podID="6cb6d8e8-20ff-44f0-a158-592cc43461ff" containerID="4a048c79dea282868940c31ed9d5983da55c2b8bf112dd199ffdb05751f27167" exitCode=0
Nov 03 06:11:22.362781 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.362720    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq" event={"ID":"6cb6d8e8-20ff-44f0-a158-592cc43461ff","Type":"ContainerDied","Data":"4a048c79dea282868940c31ed9d5983da55c2b8bf112dd199ffdb05751f27167"}
Nov 03 06:11:22.365930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.365906    4528 generic.go:356] "Generic (PLEG): container finished" podID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerID="339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470" exitCode=0
Nov 03 06:11:22.366056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.365950    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"
Nov 03 06:11:22.366056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.365988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" event={"ID":"81219190-ee1a-4ecf-acdf-c29942df3e8c","Type":"ContainerDied","Data":"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"}
Nov 03 06:11:22.366056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.366022    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg" event={"ID":"81219190-ee1a-4ecf-acdf-c29942df3e8c","Type":"ContainerDied","Data":"36659bbadd93d4d1fc9704a8a7689f5208c11d7056f944807604cf01325ed3c8"}
Nov 03 06:11:22.367983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.367961    4528 generic.go:356] "Generic (PLEG): container finished" podID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerID="8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6" exitCode=0
Nov 03 06:11:22.368117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.368100    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" event={"ID":"ee16f136-f2d3-41e7-9075-dff10ab7f195","Type":"ContainerDied","Data":"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"}
Nov 03 06:11:22.368212 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.368142    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc" event={"ID":"ee16f136-f2d3-41e7-9075-dff10ab7f195","Type":"ContainerDied","Data":"8a888a0f1fcf817314c60db63d6008db900f17787367f140aa31be93890da258"}
Nov 03 06:11:22.368272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.368219    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"
Nov 03 06:11:22.381344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.380085    4528 scope.go:122] "RemoveContainer" containerID="68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"
Nov 03 06:11:22.386592 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.383132    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe\": container with ID starting with 68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe not found: ID does not exist" containerID="68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"
Nov 03 06:11:22.386592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.383174    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe"} err="failed to get container status \"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe\": rpc error: code = NotFound desc = could not find container \"68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe\": container with ID starting with 68067f2b0d6cd7dd90dc686ff9347213a0b7e06072cfbed02df770a42174f7fe not found: ID does not exist"
Nov 03 06:11:22.386592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.383205    4528 scope.go:122] "RemoveContainer" containerID="6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"
Nov 03 06:11:22.386592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.386455    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p" event={"ID":"95980fca-5c88-40d0-9004-1609238d9708","Type":"ContainerDied","Data":"b4e75610997274e8fe0977a763828f77c24e29850a370e2b80b7bc214529bbe9"}
Nov 03 06:11:22.386592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.386582    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"
Nov 03 06:11:22.393641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.393607    4528 generic.go:356] "Generic (PLEG): container finished" podID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerID="aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3" exitCode=0
Nov 03 06:11:22.393783 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.393718    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"
Nov 03 06:11:22.394437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.394346    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"
Nov 03 06:11:22.394577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.394513    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerDied","Data":"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"}
Nov 03 06:11:22.394669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.394576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg" event={"ID":"8ac60968-6d3b-441f-90ad-04d6051d5f0f","Type":"ContainerDied","Data":"e788322c3a90a2f64895d40e2642f6ce4e6d773ad07d5df279315dd02c401718"}
Nov 03 06:11:22.407344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.407247    4528 scope.go:122] "RemoveContainer" containerID="6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"
Nov 03 06:11:22.407641 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.407623    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3\": container with ID starting with 6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3 not found: ID does not exist" containerID="6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"
Nov 03 06:11:22.407711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.407662    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3"} err="failed to get container status \"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3\": rpc error: code = NotFound desc = could not find container \"6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3\": container with ID starting with 6900af5c9486a52b91af76c118450f559ee0615d9495df16e858b12df53881b3 not found: ID does not exist"
Nov 03 06:11:22.407711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.407687    4528 scope.go:122] "RemoveContainer" containerID="339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"
Nov 03 06:11:22.423053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.423017    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:11:22.424529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.424420    4528 scope.go:122] "RemoveContainer" containerID="339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"
Nov 03 06:11:22.424765 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.424736    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470\": container with ID starting with 339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470 not found: ID does not exist" containerID="339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"
Nov 03 06:11:22.424892 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.424775    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470"} err="failed to get container status \"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470\": rpc error: code = NotFound desc = could not find container \"339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470\": container with ID starting with 339aae141f8a0c43f77931c1e4625acbb007b94cd38646ffcc96373d0c480470 not found: ID does not exist"
Nov 03 06:11:22.424892 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.424833    4528 scope.go:122] "RemoveContainer" containerID="8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"
Nov 03 06:11:22.432540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.432512    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7pp4c"]
Nov 03 06:11:22.437851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.437825    4528 scope.go:122] "RemoveContainer" containerID="8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"
Nov 03 06:11:22.438772 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.438508    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6\": container with ID starting with 8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6 not found: ID does not exist" containerID="8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"
Nov 03 06:11:22.438772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.438544    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6"} err="failed to get container status \"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6\": rpc error: code = NotFound desc = could not find container \"8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6\": container with ID starting with 8c3730f47a63525e0b80a79cad51e214cb288d9af96f232397c8af61f095a8b6 not found: ID does not exist"
Nov 03 06:11:22.438772 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.438569    4528 scope.go:122] "RemoveContainer" containerID="a265c11abb5ae06013da3d718b875c6a20d54bddc711961b3212e81469735e66"
Nov 03 06:11:22.454447 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.454423    4528 scope.go:122] "RemoveContainer" containerID="aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"
Nov 03 06:11:22.465476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.465451    4528 scope.go:122] "RemoveContainer" containerID="770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"
Nov 03 06:11:22.478414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.478389    4528 scope.go:122] "RemoveContainer" containerID="aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"
Nov 03 06:11:22.478768 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.478740    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3\": container with ID starting with aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3 not found: ID does not exist" containerID="aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"
Nov 03 06:11:22.478872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.478779    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3"} err="failed to get container status \"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3\": rpc error: code = NotFound desc = could not find container \"aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3\": container with ID starting with aba80d8c21e359e8bef0acb1a79bcb60d366758988a18724bfe08855950e14c3 not found: ID does not exist"
Nov 03 06:11:22.478872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.478848    4528 scope.go:122] "RemoveContainer" containerID="770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"
Nov 03 06:11:22.479138 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:22.479115    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312\": container with ID starting with 770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312 not found: ID does not exist" containerID="770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"
Nov 03 06:11:22.479217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.479157    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312"} err="failed to get container status \"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312\": rpc error: code = NotFound desc = could not find container \"770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312\": container with ID starting with 770fdc40fc9d1ed9357531c9e5d0055541c5507c7c6cee123f083a14f2026312 not found: ID does not exist"
Nov 03 06:11:22.615278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.615225    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:11:22.622455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.622421    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-j47lx"]
Nov 03 06:11:22.668555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.668519    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:11:22.675176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.675146    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-5c3542/distinct-2-w2-5c3542-kserve-655cf9c998-t6b6x"]
Nov 03 06:11:22.712682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.712646    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:11:22.720041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.720009    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-7b4pg"]
Nov 03 06:11:22.750360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.750327    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:11:22.760095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.760065    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-d6zcc"]
Nov 03 06:11:22.802209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.802175    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:11:22.811619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.811583    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-df7236-openshift-default-6879978f5c-brndg"]
Nov 03 06:11:22.852733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.852696    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:11:22.858447 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:22.858418    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-be1a66-openshift-default-b858d5bf5-jth2p"]
Nov 03 06:11:23.061374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.061339    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"]
Nov 03 06:11:23.067103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.067072    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"]
Nov 03 06:11:23.072983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.072955    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39cae823-j45zw"]
Nov 03 06:11:23.078131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.078104    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b35c084-xkvcd"]
Nov 03 06:11:23.568266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.568240    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:23.594770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.594742    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:23.670670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670583    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\") pod \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") "
Nov 03 06:11:23.670670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670634    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\") pod \"fb3b4a50-0498-4d01-894c-591c037e18f2\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") "
Nov 03 06:11:23.670931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670705    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\" (UniqueName: \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\") pod \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") "
Nov 03 06:11:23.670931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\" (UniqueName: \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\") pod \"fb3b4a50-0498-4d01-894c-591c037e18f2\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") "
Nov 03 06:11:23.670931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670878    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\") pod \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\" (UID: \"6cb6d8e8-20ff-44f0-a158-592cc43461ff\") "
Nov 03 06:11:23.670931 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.670924    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\") pod \"fb3b4a50-0498-4d01-894c-591c037e18f2\" (UID: \"fb3b4a50-0498-4d01-894c-591c037e18f2\") "
Nov 03 06:11:23.671485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.671444    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca" pod "6cb6d8e8-20ff-44f0-a158-592cc43461ff" (UID: "6cb6d8e8-20ff-44f0-a158-592cc43461ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:23.671740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.671698    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca" pod "fb3b4a50-0498-4d01-894c-591c037e18f2" (UID: "fb3b4a50-0498-4d01-894c-591c037e18f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:23.673705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.673672    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96" pod "fb3b4a50-0498-4d01-894c-591c037e18f2" (UID: "fb3b4a50-0498-4d01-894c-591c037e18f2"). InnerVolumeSpecName "kube-api-access-7zw96". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:23.673884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.673859    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca" pod "6cb6d8e8-20ff-44f0-a158-592cc43461ff" (UID: "6cb6d8e8-20ff-44f0-a158-592cc43461ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:23.674276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.674246    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca" pod "fb3b4a50-0498-4d01-894c-591c037e18f2" (UID: "fb3b4a50-0498-4d01-894c-591c037e18f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:23.674476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.674461    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc" pod "6cb6d8e8-20ff-44f0-a158-592cc43461ff" (UID: "6cb6d8e8-20ff-44f0-a158-592cc43461ff"). InnerVolumeSpecName "kube-api-access-2lfbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:23.749020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.748989    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" path="/var/lib/kubelet/pods/45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67/volumes"
Nov 03 06:11:23.749479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.749466    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" path="/var/lib/kubelet/pods/81219190-ee1a-4ecf-acdf-c29942df3e8c/volumes"
Nov 03 06:11:23.749932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.749917    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" path="/var/lib/kubelet/pods/8ac60968-6d3b-441f-90ad-04d6051d5f0f/volumes"
Nov 03 06:11:23.750673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.750658    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95980fca-5c88-40d0-9004-1609238d9708" path="/var/lib/kubelet/pods/95980fca-5c88-40d0-9004-1609238d9708/volumes"
Nov 03 06:11:23.751099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.751085    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" path="/var/lib/kubelet/pods/9c5589a2-bb5c-4eb0-9931-b383d8f37850/volumes"
Nov 03 06:11:23.751487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.751475    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" path="/var/lib/kubelet/pods/c220ecdf-1400-4b8b-8df1-ff68f00d8748/volumes"
Nov 03 06:11:23.752159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.752144    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9a89d27-9c98-4e2f-8271-a5898257d53f" path="/var/lib/kubelet/pods/c9a89d27-9c98-4e2f-8271-a5898257d53f/volumes"
Nov 03 06:11:23.752538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.752525    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5a3755a-8c97-4480-806b-703483a7b5c5" path="/var/lib/kubelet/pods/d5a3755a-8c97-4480-806b-703483a7b5c5/volumes"
Nov 03 06:11:23.752952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.752939    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" path="/var/lib/kubelet/pods/ee16f136-f2d3-41e7-9075-dff10ab7f195/volumes"
Nov 03 06:11:23.772715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772692    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:23.772715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772715    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:23.772875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772725    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lfbc\" (UniqueName: \"kubernetes.io/projected/6cb6d8e8-20ff-44f0-a158-592cc43461ff-kube-api-access-2lfbc\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:23.772875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772734    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zw96\" (UniqueName: \"kubernetes.io/projected/fb3b4a50-0498-4d01-894c-591c037e18f2-kube-api-access-7zw96\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:23.772875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772743    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb6d8e8-20ff-44f0-a158-592cc43461ff-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:23.772875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:23.772752    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb3b4a50-0498-4d01-894c-591c037e18f2-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:24.424546 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.424505    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl" event={"ID":"fb3b4a50-0498-4d01-894c-591c037e18f2","Type":"ContainerDied","Data":"163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8"}
Nov 03 06:11:24.424719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.424573    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="163aac15915aacf373bd712ed21a9866d627e55d4d7df969b1273a43cf5b6be8"
Nov 03 06:11:24.424719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.424514    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"
Nov 03 06:11:24.426242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.426222    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"
Nov 03 06:11:24.426354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.426271    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq" event={"ID":"6cb6d8e8-20ff-44f0-a158-592cc43461ff","Type":"ContainerDied","Data":"4855e4dfe832c2bd66408487f7d54c2562cd60884e596706bc216c65198f78e5"}
Nov 03 06:11:24.426354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.426310    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4855e4dfe832c2bd66408487f7d54c2562cd60884e596706bc216c65198f78e5"
Nov 03 06:11:24.790186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:24.790092    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main" containerID="cri-o://5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50" gracePeriod=45
Nov 03 06:11:25.054266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.054240    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188042    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188104    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188120    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188157    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\" (UniqueName: \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188293    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\" (UniqueName: \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188333    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\") pod \"c309c4d0-3513-4c7b-991d-88c891e6ce54\" (UID: \"c309c4d0-3513-4c7b-991d-88c891e6ce54\") "
Nov 03 06:11:25.188830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188767    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:25.189538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.188993    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:25.189538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.189101    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:25.193486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.193433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:11:25.193689 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.193537    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:25.195084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.195055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75" pod "c309c4d0-3513-4c7b-991d-88c891e6ce54" (UID: "c309c4d0-3513-4c7b-991d-88c891e6ce54"). InnerVolumeSpecName "kube-api-access-n7z75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:25.290017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.289970    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.290017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.290011    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.290017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.290025    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.290318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.290038    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c309c4d0-3513-4c7b-991d-88c891e6ce54-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.290318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.290051    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7z75\" (UniqueName: \"kubernetes.io/projected/c309c4d0-3513-4c7b-991d-88c891e6ce54-kube-api-access-n7z75\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.290318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.290065    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c309c4d0-3513-4c7b-991d-88c891e6ce54-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:25.438997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.438911    4528 generic.go:356] "Generic (PLEG): container finished" podID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerID="5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50" exitCode=0
Nov 03 06:11:25.438997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.438981    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"
Nov 03 06:11:25.439218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.438994    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" event={"ID":"c309c4d0-3513-4c7b-991d-88c891e6ce54","Type":"ContainerDied","Data":"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"}
Nov 03 06:11:25.439218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.439038    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h" event={"ID":"c309c4d0-3513-4c7b-991d-88c891e6ce54","Type":"ContainerDied","Data":"78feb78cabcfe33ca3e18b41ea1f6af9e51ca5e13b4fb56ec39f71232eb57df7"}
Nov 03 06:11:25.439218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.439061    4528 scope.go:122] "RemoveContainer" containerID="5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"
Nov 03 06:11:25.450553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.450531    4528 scope.go:122] "RemoveContainer" containerID="5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"
Nov 03 06:11:25.450918 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:25.450893    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50\": container with ID starting with 5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50 not found: ID does not exist" containerID="5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"
Nov 03 06:11:25.450998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.450931    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50"} err="failed to get container status \"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50\": rpc error: code = NotFound desc = could not find container \"5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50\": container with ID starting with 5bba15126e245f60f9f85b76106be9a6d15f7ff8c9eed997c011f668ed321c50 not found: ID does not exist"
Nov 03 06:11:25.501446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.501405    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:11:25.517824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.515147    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-9688d8/distinct-2-w1-9688d8-kserve-6f87f67c84-h2r6h"]
Nov 03 06:11:25.800866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:25.793595    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" path="/var/lib/kubelet/pods/c309c4d0-3513-4c7b-991d-88c891e6ce54/volumes"
Nov 03 06:11:26.033227 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:26.033191    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"]
Nov 03 06:11:26.042419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:26.042386    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-5c3542-7w6vt"]
Nov 03 06:11:27.299022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:27.298984    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:27.299393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:27.298996    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:27.748742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:27.748712    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75569012-1c4e-45ea-a1f5-42d09d184a07" path="/var/lib/kubelet/pods/75569012-1c4e-45ea-a1f5-42d09d184a07/volumes"
Nov 03 06:11:30.408324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:30.408280    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:32.846150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.846108    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:11:32.846615 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.846325    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy" containerID="cri-o://3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236" gracePeriod=30
Nov 03 06:11:32.858132 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.858096    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:11:32.858357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.858331    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" containerID="cri-o://2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70" gracePeriod=30
Nov 03 06:11:32.867303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.867274    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:11:32.867480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:32.867463    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy" containerID="cri-o://9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463" gracePeriod=30
Nov 03 06:11:34.447477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:34.446876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"]
Nov 03 06:11:34.454631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:34.454599    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-9688d8-gjbgv"]
Nov 03 06:11:35.059909 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:35.059858    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"]
Nov 03 06:11:35.077642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:35.077603    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-705e06bf-htm4b"]
Nov 03 06:11:35.759485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:35.759446    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aae8124-635d-43fb-8137-42894ae3663e" path="/var/lib/kubelet/pods/1aae8124-635d-43fb-8137-42894ae3663e/volumes"
Nov 03 06:11:35.760122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:35.760099    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a70bce6-ee8b-4fd5-9400-d8f8944926d8" path="/var/lib/kubelet/pods/9a70bce6-ee8b-4fd5-9400-d8f8944926d8/volumes"
Nov 03 06:11:36.042084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.041995    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"]
Nov 03 06:11:36.047776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.047748    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dffa33-bntvv"]
Nov 03 06:11:36.364540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.364499    4528 patch_prober.go:29] interesting pod/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:36.364689 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.364574    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:36.938764 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.938739    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:11:36.963352 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:36.963325    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:11:37.045863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.045822    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\" (UniqueName: \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\") pod \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") "
Nov 03 06:11:37.047276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.046605    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\") pod \"831d6c36-a2ab-4013-b701-2206555582f5\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") "
Nov 03 06:11:37.047276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.046818    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\") pod \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\" (UID: \"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f\") "
Nov 03 06:11:37.047276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.046858    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\" (UniqueName: \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\") pod \"831d6c36-a2ab-4013-b701-2206555582f5\" (UID: \"831d6c36-a2ab-4013-b701-2206555582f5\") "
Nov 03 06:11:37.049436 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.048256    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume" pod "70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" (UID: "70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:37.051011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.049807    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume" pod "831d6c36-a2ab-4013-b701-2206555582f5" (UID: "831d6c36-a2ab-4013-b701-2206555582f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:37.059630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.059472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj" pod "831d6c36-a2ab-4013-b701-2206555582f5" (UID: "831d6c36-a2ab-4013-b701-2206555582f5"). InnerVolumeSpecName "kube-api-access-862qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:37.061416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.061382    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9" pod "70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" (UID: "70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f"). InnerVolumeSpecName "kube-api-access-vfvz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:37.061536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.061440    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"]
Nov 03 06:11:37.068947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.068915    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-085a34ca-mz9fw"]
Nov 03 06:11:37.149071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.148992    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfvz9\" (UniqueName: \"kubernetes.io/projected/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-kube-api-access-vfvz9\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:37.149071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.149022    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/831d6c36-a2ab-4013-b701-2206555582f5-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:37.149071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.149034    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:37.149071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.149042    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-862qj\" (UniqueName: \"kubernetes.io/projected/831d6c36-a2ab-4013-b701-2206555582f5-kube-api-access-862qj\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:37.579673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.579636    4528 generic.go:356] "Generic (PLEG): container finished" podID="831d6c36-a2ab-4013-b701-2206555582f5" containerID="c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0" exitCode=0
Nov 03 06:11:37.579880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.579705    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"
Nov 03 06:11:37.579880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.579722    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" event={"ID":"831d6c36-a2ab-4013-b701-2206555582f5","Type":"ContainerDied","Data":"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"}
Nov 03 06:11:37.579880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.579840    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs" event={"ID":"831d6c36-a2ab-4013-b701-2206555582f5","Type":"ContainerDied","Data":"5b9e5f56eeb223272d4f7a9a75b3b0a0ba8d3de394d6f460575c35e0a1208c06"}
Nov 03 06:11:37.579880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.579872    4528 scope.go:122] "RemoveContainer" containerID="c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"
Nov 03 06:11:37.581418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.581393    4528 generic.go:356] "Generic (PLEG): container finished" podID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerID="58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61" exitCode=0
Nov 03 06:11:37.581528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.581444    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"
Nov 03 06:11:37.581528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.581465    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" event={"ID":"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f","Type":"ContainerDied","Data":"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"}
Nov 03 06:11:37.581528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.581509    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x" event={"ID":"70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f","Type":"ContainerDied","Data":"a6d434f6643b2ce9bf30ffab207a0c17d36efd7c63057fa9918b86ce31c47343"}
Nov 03 06:11:37.592660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.592638    4528 scope.go:122] "RemoveContainer" containerID="c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"
Nov 03 06:11:37.592983 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:37.592961    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0\": container with ID starting with c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0 not found: ID does not exist" containerID="c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"
Nov 03 06:11:37.593047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.592991    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0"} err="failed to get container status \"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0\": rpc error: code = NotFound desc = could not find container \"c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0\": container with ID starting with c66a9fd2d37f1b96dfcd35e85314e6693b00ec3dbb16a072e71bcac5690718c0 not found: ID does not exist"
Nov 03 06:11:37.593047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.593010    4528 scope.go:122] "RemoveContainer" containerID="58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"
Nov 03 06:11:37.603534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.603516    4528 scope.go:122] "RemoveContainer" containerID="58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"
Nov 03 06:11:37.603827 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:37.603785    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61\": container with ID starting with 58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61 not found: ID does not exist" containerID="58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"
Nov 03 06:11:37.603905 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.603829    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61"} err="failed to get container status \"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61\": rpc error: code = NotFound desc = could not find container \"58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61\": container with ID starting with 58c291c506fa747cb021a24a28153c56a99d54ea11581a7634d3980edb925e61 not found: ID does not exist"
Nov 03 06:11:37.645830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.645786    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:11:37.657803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.657768    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-5c3542-c4ddd876c-dcjqs"]
Nov 03 06:11:37.686341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.686301    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:11:37.698093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.698068    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-5c3542-76c5d4b6f9-fbc4x"]
Nov 03 06:11:37.748340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.748298    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1402809e-9c66-445b-8d96-addd2cfed64f" path="/var/lib/kubelet/pods/1402809e-9c66-445b-8d96-addd2cfed64f/volumes"
Nov 03 06:11:37.748859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.748839    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d2d3b31-7a0e-40ae-a1b8-33b588b593d2" path="/var/lib/kubelet/pods/3d2d3b31-7a0e-40ae-a1b8-33b588b593d2/volumes"
Nov 03 06:11:37.749260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.749242    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" path="/var/lib/kubelet/pods/70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f/volumes"
Nov 03 06:11:37.750056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:37.750033    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="831d6c36-a2ab-4013-b701-2206555582f5" path="/var/lib/kubelet/pods/831d6c36-a2ab-4013-b701-2206555582f5/volumes"
Nov 03 06:11:38.136562 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.136367    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:11:38.163452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.163429    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:11:38.181096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.181076    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:11:38.261629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261592    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.261830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261661    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.261830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.261830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261727    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.261830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261763    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261884    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261946    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261953    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.261978    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262007    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262037    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262086    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262114    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262142    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262169    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262181    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262258    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\") pod \"90d05fa7-9d57-46f6-a0ef-afea586291bd\" (UID: \"90d05fa7-9d57-46f6-a0ef-afea586291bd\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262284    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\") pod \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\" (UID: \"8acc50f4-c6c7-412b-bdea-70f82ba0186f\") "
Nov 03 06:11:38.262487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262346    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.262891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262503    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:38.262891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.262842    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:38.263069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263044    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.263138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263088    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.263138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263106    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.263138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263122    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d05fa7-9d57-46f6-a0ef-afea586291bd-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.263138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263135    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.263338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263148    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.263398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263340    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.263727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263643    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.263867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.263773    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.264159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.264124    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.265342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.265297    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.265711 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.265689    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "kube-api-access-knlh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.266412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.266369    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:38.266530 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.266506    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.266952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.266922    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:38.267513 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.267486    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.267960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.267935    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq" pod "8acc50f4-c6c7-412b-bdea-70f82ba0186f" (UID: "8acc50f4-c6c7-412b-bdea-70f82ba0186f"). InnerVolumeSpecName "kube-api-access-gl2kq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.268055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.268034    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token" pod "90d05fa7-9d57-46f6-a0ef-afea586291bd" (UID: "90d05fa7-9d57-46f6-a0ef-afea586291bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.363541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363500    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.363541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363541    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.363843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363582    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.363843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363629    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.363843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363657    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.363843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363687    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.364069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363844    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.364069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363913    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.364069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363933    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.364069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.363993    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\") pod \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\" (UID: \"5db0f042-5cdf-43a2-a56e-a44116bd2e04\") "
Nov 03 06:11:38.364069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364026    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.364327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364283    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364466    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364487    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knlh4\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-kube-api-access-knlh4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364511    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364526    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364539    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364554    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364570    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl2kq\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-kube-api-access-gl2kq\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364584    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364597    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364610    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364623    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364635    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364646    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364660    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364673    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364686    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d05fa7-9d57-46f6-a0ef-afea586291bd-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364699    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8acc50f4-c6c7-412b-bdea-70f82ba0186f-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.364744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.364708    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.366084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.366059    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:38.366243 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.366226    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:38.366637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.366623    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.366785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.366771    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm" pod "5db0f042-5cdf-43a2-a56e-a44116bd2e04" (UID: "5db0f042-5cdf-43a2-a56e-a44116bd2e04"). InnerVolumeSpecName "kube-api-access-rflbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465530    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rflbm\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-kube-api-access-rflbm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465563    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465573    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465581    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465590    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5db0f042-5cdf-43a2-a56e-a44116bd2e04-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.465611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.465598    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5db0f042-5cdf-43a2-a56e-a44116bd2e04-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:38.592478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.592445    4528 generic.go:356] "Generic (PLEG): container finished" podID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerID="2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70" exitCode=0
Nov 03 06:11:38.592668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.592490    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"
Nov 03 06:11:38.592668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.592546    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" event={"ID":"8acc50f4-c6c7-412b-bdea-70f82ba0186f","Type":"ContainerDied","Data":"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"}
Nov 03 06:11:38.592668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.592613    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz" event={"ID":"8acc50f4-c6c7-412b-bdea-70f82ba0186f","Type":"ContainerDied","Data":"4629bcf4e2f05caa5a238f5e62392622edd14d09d5899274fbca41b735e7aba6"}
Nov 03 06:11:38.592668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.592644    4528 scope.go:122] "RemoveContainer" containerID="2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"
Nov 03 06:11:38.595523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.595499    4528 generic.go:356] "Generic (PLEG): container finished" podID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerID="9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463" exitCode=0
Nov 03 06:11:38.595636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.595551    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"
Nov 03 06:11:38.595636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.595585    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" event={"ID":"5db0f042-5cdf-43a2-a56e-a44116bd2e04","Type":"ContainerDied","Data":"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"}
Nov 03 06:11:38.595757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.595637    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn" event={"ID":"5db0f042-5cdf-43a2-a56e-a44116bd2e04","Type":"ContainerDied","Data":"35acc0515d6a6299a09ebc372debd18efb16b91a40c3c43f5f223f70ef9b7eba"}
Nov 03 06:11:38.599307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.599282    4528 generic.go:356] "Generic (PLEG): container finished" podID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerID="3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236" exitCode=0
Nov 03 06:11:38.599422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.599350    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerDied","Data":"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"}
Nov 03 06:11:38.599422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.599391    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57" event={"ID":"90d05fa7-9d57-46f6-a0ef-afea586291bd","Type":"ContainerDied","Data":"d21ec1a8e9e43553517cdd4355807c2873d8c5f467a8baa2a54a56b5918f10c5"}
Nov 03 06:11:38.599422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.599351    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"
Nov 03 06:11:38.608827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.608785    4528 scope.go:122] "RemoveContainer" containerID="2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"
Nov 03 06:11:38.609133 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:38.609104    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70\": container with ID starting with 2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70 not found: ID does not exist" containerID="2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"
Nov 03 06:11:38.609238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.609148    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70"} err="failed to get container status \"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70\": rpc error: code = NotFound desc = could not find container \"2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70\": container with ID starting with 2d400c9f77b90c30bc85cd2ad66483087b05cfb5e1f505438bf5ec6bc7c9ba70 not found: ID does not exist"
Nov 03 06:11:38.609238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.609175    4528 scope.go:122] "RemoveContainer" containerID="9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"
Nov 03 06:11:38.623542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.623519    4528 scope.go:122] "RemoveContainer" containerID="9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"
Nov 03 06:11:38.623870 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:38.623845    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463\": container with ID starting with 9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463 not found: ID does not exist" containerID="9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"
Nov 03 06:11:38.623955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.623879    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463"} err="failed to get container status \"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463\": rpc error: code = NotFound desc = could not find container \"9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463\": container with ID starting with 9f5d95845b8f5f679bd7562507e0fdbfaa1c14fff714bbc66298b5bef41f2463 not found: ID does not exist"
Nov 03 06:11:38.623955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.623903    4528 scope.go:122] "RemoveContainer" containerID="3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"
Nov 03 06:11:38.650603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.650578    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:11:38.655873 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.655847    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8r7mz"]
Nov 03 06:11:38.655962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.655853    4528 scope.go:122] "RemoveContainer" containerID="72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"
Nov 03 06:11:38.666392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.666362    4528 scope.go:122] "RemoveContainer" containerID="3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"
Nov 03 06:11:38.666647 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:38.666628    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236\": container with ID starting with 3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236 not found: ID does not exist" containerID="3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"
Nov 03 06:11:38.666691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.666656    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236"} err="failed to get container status \"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236\": rpc error: code = NotFound desc = could not find container \"3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236\": container with ID starting with 3450832e2f166aed6c7e716b5bec50482ce6435a90f4231b33f6626c6e3a4236 not found: ID does not exist"
Nov 03 06:11:38.666691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.666674    4528 scope.go:122] "RemoveContainer" containerID="72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"
Nov 03 06:11:38.666959 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:38.666936    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143\": container with ID starting with 72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143 not found: ID does not exist" containerID="72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"
Nov 03 06:11:38.667000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.666967    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143"} err="failed to get container status \"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143\": rpc error: code = NotFound desc = could not find container \"72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143\": container with ID starting with 72aca4a999c7fe247513c7264b88cb9abd5e1a8777a3c5f9fc0dde603c008143 not found: ID does not exist"
Nov 03 06:11:38.711373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.711341    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:11:38.717411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.717342    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-8cmnn"]
Nov 03 06:11:38.752156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.752117    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:11:38.757050 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:38.757025    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-5c3542-openshift-default-754655f44f-2dm57"]
Nov 03 06:11:39.038665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.038584    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"]
Nov 03 06:11:39.046630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.046602    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d1f8b40-pl44d"]
Nov 03 06:11:39.754133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.754093    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14494674-4433-4e53-b7bf-678bb16d5e70" path="/var/lib/kubelet/pods/14494674-4433-4e53-b7bf-678bb16d5e70/volumes"
Nov 03 06:11:39.754750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.754731    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" path="/var/lib/kubelet/pods/5db0f042-5cdf-43a2-a56e-a44116bd2e04/volumes"
Nov 03 06:11:39.755298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.755283    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" path="/var/lib/kubelet/pods/8acc50f4-c6c7-412b-bdea-70f82ba0186f/volumes"
Nov 03 06:11:39.756351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:39.756329    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" path="/var/lib/kubelet/pods/90d05fa7-9d57-46f6-a0ef-afea586291bd/volumes"
Nov 03 06:11:40.407249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:40.407206    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:11:40.926754 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:40.926684    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:11:41.091955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.091913    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\") pod \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") "
Nov 03 06:11:41.092168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.091968    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\" (UniqueName: \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\") pod \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\" (UID: \"789d295c-7bd4-4bb9-9f9d-bc56d6102849\") "
Nov 03 06:11:41.092391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.092363    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume" pod "789d295c-7bd4-4bb9-9f9d-bc56d6102849" (UID: "789d295c-7bd4-4bb9-9f9d-bc56d6102849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:41.094134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.094111    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx" pod "789d295c-7bd4-4bb9-9f9d-bc56d6102849" (UID: "789d295c-7bd4-4bb9-9f9d-bc56d6102849"). InnerVolumeSpecName "kube-api-access-c5jdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:41.193860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.193754    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/789d295c-7bd4-4bb9-9f9d-bc56d6102849-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:41.193860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.193804    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5jdx\" (UniqueName: \"kubernetes.io/projected/789d295c-7bd4-4bb9-9f9d-bc56d6102849-kube-api-access-c5jdx\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:41.636593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.636555    4528 generic.go:356] "Generic (PLEG): container finished" podID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerID="4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2" exitCode=0
Nov 03 06:11:41.636771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.636606    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"
Nov 03 06:11:41.636771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.636641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" event={"ID":"789d295c-7bd4-4bb9-9f9d-bc56d6102849","Type":"ContainerDied","Data":"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"}
Nov 03 06:11:41.636771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.636701    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz" event={"ID":"789d295c-7bd4-4bb9-9f9d-bc56d6102849","Type":"ContainerDied","Data":"0dce9aafd7c59dd537314042a57fa0756497d5de9458b6b63333f056c121e096"}
Nov 03 06:11:41.636771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.636724    4528 scope.go:122] "RemoveContainer" containerID="4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"
Nov 03 06:11:41.649175 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.649155    4528 scope.go:122] "RemoveContainer" containerID="4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"
Nov 03 06:11:41.649493 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:41.649468    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2\": container with ID starting with 4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2 not found: ID does not exist" containerID="4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"
Nov 03 06:11:41.649575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.649503    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2"} err="failed to get container status \"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2\": rpc error: code = NotFound desc = could not find container \"4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2\": container with ID starting with 4be07243090b14d3deb10b1168879fab995a9be0d199e4e68c2c02d4b062fcd2 not found: ID does not exist"
Nov 03 06:11:41.696976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.696938    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:11:41.709140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.709106    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-9688d8-85cdcf696c-9wqzz"]
Nov 03 06:11:41.748923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:41.748890    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" path="/var/lib/kubelet/pods/789d295c-7bd4-4bb9-9f9d-bc56d6102849/volumes"
Nov 03 06:11:42.409367 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.409332    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:11:42.409869 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.409576    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" containerID="cri-o://54dfdb2772861492384834b8f69b685d3338e9488c54814626bf38252349933b" gracePeriod=30
Nov 03 06:11:42.420602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.420563    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:11:42.420814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.420777    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" containerID="cri-o://8d90ab093b2dc2211301cd1e4ca7618ab1cad0ea284743e1ce7c27e7af33ff6a" gracePeriod=30
Nov 03 06:11:42.433444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.433415    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:11:42.433621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:42.433601    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" containerID="cri-o://81813a202d47385b67754eb4c58ec5ae6b469c6e861c79e2cafcc4c6c0045213" gracePeriod=30
Nov 03 06:11:43.692736 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:11:43.692695    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:11:46.570342 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:46.570297    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:46.570733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:46.570382    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:47.018161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.018076    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:47.018161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.018146    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:47.373424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.373387    4528 patch_prober.go:29] interesting pod/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:11:47.373601 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.373470    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:11:47.699819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.699438    4528 generic.go:356] "Generic (PLEG): container finished" podID="a75905cf-8c83-4133-a545-bcd5e4027069" containerID="8d90ab093b2dc2211301cd1e4ca7618ab1cad0ea284743e1ce7c27e7af33ff6a" exitCode=0
Nov 03 06:11:47.699819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.699545    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" event={"ID":"a75905cf-8c83-4133-a545-bcd5e4027069","Type":"ContainerDied","Data":"8d90ab093b2dc2211301cd1e4ca7618ab1cad0ea284743e1ce7c27e7af33ff6a"}
Nov 03 06:11:47.702108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.702055    4528 generic.go:356] "Generic (PLEG): container finished" podID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerID="81813a202d47385b67754eb4c58ec5ae6b469c6e861c79e2cafcc4c6c0045213" exitCode=0
Nov 03 06:11:47.702253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.702111    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" event={"ID":"6cebc3ba-dad7-4210-bea9-0bd43834b6af","Type":"ContainerDied","Data":"81813a202d47385b67754eb4c58ec5ae6b469c6e861c79e2cafcc4c6c0045213"}
Nov 03 06:11:47.718239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.718209    4528 generic.go:356] "Generic (PLEG): container finished" podID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerID="54dfdb2772861492384834b8f69b685d3338e9488c54814626bf38252349933b" exitCode=0
Nov 03 06:11:47.718455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.718378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" event={"ID":"0f0dbecd-a3c7-4168-a28a-8374e88b13f1","Type":"ContainerDied","Data":"54dfdb2772861492384834b8f69b685d3338e9488c54814626bf38252349933b"}
Nov 03 06:11:47.721230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.720865    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:11:47.769045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.769017    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:11:47.772903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.772843    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:11:47.778015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.777988    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778050    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778078    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778144    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778204    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778311    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778392    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778403    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:47.778532 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778478    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\") pod \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\" (UID: \"0f0dbecd-a3c7-4168-a28a-8374e88b13f1\") "
Nov 03 06:11:47.778707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778513    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.778707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778590    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.778893 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778755    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.778997 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778973    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.779058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.778992    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.779058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.779016    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.779058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.779029    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.779058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.779046    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.781345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.781309    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:47.781473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.781393    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.782073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.782046    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "kube-api-access-bv2c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.784679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.784646    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy" pod "0f0dbecd-a3c7-4168-a28a-8374e88b13f1" (UID: "0f0dbecd-a3c7-4168-a28a-8374e88b13f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.880281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880241    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880300    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880338    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880366    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880406    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880437    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.880492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880488    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880518    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880588    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880640    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880616    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880671    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880706    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880707    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.880822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880787    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880850    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880887    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880916    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880945    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\") pod \"a75905cf-8c83-4133-a545-bcd5e4027069\" (UID: \"a75905cf-8c83-4133-a545-bcd5e4027069\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880989    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\") pod \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\" (UID: \"6cebc3ba-dad7-4210-bea9-0bd43834b6af\") "
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880780    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:47.881226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.880668    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.881664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.881195    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.881664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.881208    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.881664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.881492    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.883472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883443    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:47.883472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883459    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "kube-api-access-mxfxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.883854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883824    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.883958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883881    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.883958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883901    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:11:47.884089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883990    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:11:47.884089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.883874    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.884256 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884235    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884261    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884277    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884289    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884301    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884314    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv2c5\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-kube-api-access-bv2c5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884328    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884340    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884352    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884365    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884398    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884413    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a75905cf-8c83-4133-a545-bcd5e4027069-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884426    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884457    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884470    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxfxj\" (UniqueName: \"kubernetes.io/projected/6cebc3ba-dad7-4210-bea9-0bd43834b6af-kube-api-access-mxfxj\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884485    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f0dbecd-a3c7-4168-a28a-8374e88b13f1-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884513    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884526    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a75905cf-8c83-4133-a545-bcd5e4027069-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884542    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.884688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.884556    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a75905cf-8c83-4133-a545-bcd5e4027069-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.887202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.887172    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.887334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.887229    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f" pod "a75905cf-8c83-4133-a545-bcd5e4027069" (UID: "a75905cf-8c83-4133-a545-bcd5e4027069"). InnerVolumeSpecName "kube-api-access-cvd5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:11:47.887696 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.887667    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy" pod "6cebc3ba-dad7-4210-bea9-0bd43834b6af" (UID: "6cebc3ba-dad7-4210-bea9-0bd43834b6af"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:11:47.986071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.985990    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6cebc3ba-dad7-4210-bea9-0bd43834b6af-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.986071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.986020    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:47.986071 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:47.986030    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvd5f\" (UniqueName: \"kubernetes.io/projected/a75905cf-8c83-4133-a545-bcd5e4027069-kube-api-access-cvd5f\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:11:48.730307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.730220    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"
Nov 03 06:11:48.730846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.730233    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8" event={"ID":"a75905cf-8c83-4133-a545-bcd5e4027069","Type":"ContainerDied","Data":"259a6155d9c9c4ca443dc96ff51d424e9ccae79590bc37b209380be69dbaee7a"}
Nov 03 06:11:48.731014 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.730903    4528 scope.go:122] "RemoveContainer" containerID="8d90ab093b2dc2211301cd1e4ca7618ab1cad0ea284743e1ce7c27e7af33ff6a"
Nov 03 06:11:48.732114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.732091    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"
Nov 03 06:11:48.732962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.732106    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx" event={"ID":"6cebc3ba-dad7-4210-bea9-0bd43834b6af","Type":"ContainerDied","Data":"14ca1742475c164b1e8599f2fc6cdfd32ebaa3788de38a522fde77a6ffd17ab7"}
Nov 03 06:11:48.734826 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.734778    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w" event={"ID":"0f0dbecd-a3c7-4168-a28a-8374e88b13f1","Type":"ContainerDied","Data":"0d6c443fdf184d5ccf4867fc82d7d3ecdc6709f4bc43a5cb1de0970ea66d3fd1"}
Nov 03 06:11:48.735064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.734909    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"
Nov 03 06:11:48.746324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.746301    4528 scope.go:122] "RemoveContainer" containerID="81813a202d47385b67754eb4c58ec5ae6b469c6e861c79e2cafcc4c6c0045213"
Nov 03 06:11:48.758549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.758527    4528 scope.go:122] "RemoveContainer" containerID="54dfdb2772861492384834b8f69b685d3338e9488c54814626bf38252349933b"
Nov 03 06:11:48.798282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.798250    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:11:48.809857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.809820    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-skml8"]
Nov 03 06:11:48.844572 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.844540    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:11:48.849640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.849612    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-cr75w"]
Nov 03 06:11:48.891407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.891362    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:11:48.896477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:48.896449    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-9688d8-openshift-default-bf4df7489-8fgpx"]
Nov 03 06:11:49.041917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.041828    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"]
Nov 03 06:11:49.050730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.050698    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d10695c-7hg5f"]
Nov 03 06:11:49.747940 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.747909    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" path="/var/lib/kubelet/pods/0f0dbecd-a3c7-4168-a28a-8374e88b13f1/volumes"
Nov 03 06:11:49.748541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.748524    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" path="/var/lib/kubelet/pods/6cebc3ba-dad7-4210-bea9-0bd43834b6af/volumes"
Nov 03 06:11:49.748971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.748957    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" path="/var/lib/kubelet/pods/a75905cf-8c83-4133-a545-bcd5e4027069/volumes"
Nov 03 06:11:49.749652 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:49.749638    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d" path="/var/lib/kubelet/pods/f1a85e9e-f9be-4ec3-bcda-358cf1ecc22d/volumes"
Nov 03 06:11:52.701934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:52.701887    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:11:52.702424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:52.701957    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:11:52.866948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:52.866907    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:11:52.867304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:52.867264    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:11:55.035466 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:55.035428    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"]
Nov 03 06:11:55.045983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:55.045955    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bebf5711-n7jd5"]
Nov 03 06:11:55.748034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:55.747999    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9632817e-c0d0-4892-977c-8c4d8acbea77" path="/var/lib/kubelet/pods/9632817e-c0d0-4892-977c-8c4d8acbea77/volumes"
Nov 03 06:11:56.032319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:56.032230    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"]
Nov 03 06:11:56.045277 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:56.045243    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f828729-gz62x"]
Nov 03 06:11:56.050116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:56.050087    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"]
Nov 03 06:11:56.055285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:56.055256    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0e26fdc4-tzfdd"]
Nov 03 06:11:57.748525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:57.748490    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e0d557-008b-4ccc-8b83-9880d9b65ca2" path="/var/lib/kubelet/pods/40e0d557-008b-4ccc-8b83-9880d9b65ca2/volumes"
Nov 03 06:11:57.748974 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:57.748953    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5eead744-165b-4b91-94b9-8f389f100f11" path="/var/lib/kubelet/pods/5eead744-165b-4b91-94b9-8f389f100f11/volumes"
Nov 03 06:11:58.033091 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:58.033007    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"]
Nov 03 06:11:58.044465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:58.044435    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79ab5ecb-8wjts"]
Nov 03 06:11:59.747893 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:11:59.747857    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5dd847f-72bd-4a71-87af-1032f3dd0313" path="/var/lib/kubelet/pods/e5dd847f-72bd-4a71-87af-1032f3dd0313/volumes"
Nov 03 06:12:00.039248 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:00.039164    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"]
Nov 03 06:12:00.050187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:00.050155    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2243a7ae-rzvgn"]
Nov 03 06:12:01.748714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:01.748683    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed290cc-5cf2-4c52-a083-b96d9cee9cd2" path="/var/lib/kubelet/pods/eed290cc-5cf2-4c52-a083-b96d9cee9cd2/volumes"
Nov 03 06:12:08.030481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.030442    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"]
Nov 03 06:12:08.031758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031726    4528 cpu_manager.go:414] "containerMap: removing container" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main"
Nov 03 06:12:08.031758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031758    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031777    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031785    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031828    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031836    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031861    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031869    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031880    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031887    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031904    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031912    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031922    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031937    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031948    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy"
Nov 03 06:12:08.031963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031957    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031983    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.031991    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032017    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032024    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032042    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032049    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032064    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032071    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032089    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032096    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032115    4528 cpu_manager.go:414] "containerMap: removing container" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032122    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032148    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032155    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032177    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fb3b4a50-0498-4d01-894c-591c037e18f2" containerName="revoke-keys"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032183    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb3b4a50-0498-4d01-894c-591c037e18f2" containerName="revoke-keys"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032203    4528 cpu_manager.go:414] "containerMap: removing container" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032210    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032230    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032237    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032251    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6cb6d8e8-20ff-44f0-a158-592cc43461ff" containerName="revoke-keys"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032258    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb6d8e8-20ff-44f0-a158-592cc43461ff" containerName="revoke-keys"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032281    4528 cpu_manager.go:414] "containerMap: removing container" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032288    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032305    4528 cpu_manager.go:414] "containerMap: removing container" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032312    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032326    4528 cpu_manager.go:414] "containerMap: removing container" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032333    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032354    4528 cpu_manager.go:414] "containerMap: removing container" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032360    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032372    4528 cpu_manager.go:414] "containerMap: removing container" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032379    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032405    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032412    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032430    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy"
Nov 03 06:12:08.032597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032436    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032704    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032728    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="70b3b9d4-9ce2-4f40-9ea0-a56f7def3b5f" containerName="payload-pre-processing"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032737    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="831d6c36-a2ab-4013-b701-2206555582f5" containerName="payload-processing"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032747    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5db0f042-5cdf-43a2-a56e-a44116bd2e04" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032761    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c03e07e-1e7a-4ab6-a890-0a379fc2ad57" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032772    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="45a0c545-7f6b-4c2c-86fa-2e6ef2ed4c67" containerName="main"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032780    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee16f136-f2d3-41e7-9075-dff10ab7f195" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032841    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8acc50f4-c6c7-412b-bdea-70f82ba0186f" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032852    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032866    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75905cf-8c83-4133-a545-bcd5e4027069" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032873    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032881    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c309c4d0-3513-4c7b-991d-88c891e6ce54" containerName="main"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032891    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b9b9c0-cf19-439f-aaad-6b654d2b2768" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032908    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea3d0c16-444e-495c-8208-b505170d2aef" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032926    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cebc3ba-dad7-4210-bea9-0bd43834b6af" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032942    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3144501e-3a21-4303-9bbc-afad4fc6dafa" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032952    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ac60968-6d3b-441f-90ad-04d6051d5f0f" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032968    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f0dbecd-a3c7-4168-a28a-8374e88b13f1" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032984    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="81219190-ee1a-4ecf-acdf-c29942df3e8c" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.032994    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c5589a2-bb5c-4eb0-9931-b383d8f37850" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.033001    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb3b4a50-0498-4d01-894c-591c037e18f2" containerName="revoke-keys"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.033016    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="789d295c-7bd4-4bb9-9f9d-bc56d6102849" containerName="payload-processing"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.033024    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c220ecdf-1400-4b8b-8df1-ff68f00d8748" containerName="istio-proxy"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.033031    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb6d8e8-20ff-44f0-a158-592cc43461ff" containerName="revoke-keys"
Nov 03 06:12:08.034233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.033046    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="95980fca-5c88-40d0-9004-1609238d9708" containerName="istio-proxy"
Nov 03 06:12:08.043879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.043845    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.045915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.045890    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"]
Nov 03 06:12:08.055568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.055534    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f754b51d-nrpvk"]
Nov 03 06:12:08.058701 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.058678    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:12:08.058962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.058939    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:12:08.060818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.060769    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"]
Nov 03 06:12:08.060922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.060859    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:12:08.060922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.060857    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:12:08.166716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.166675    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwr9m\" (UniqueName: \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.166716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.166719    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.166956 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.166840    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.267427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.267374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.267629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.267524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwr9m\" (UniqueName: \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.267629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.267559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.268330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.268303    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.269947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.269925    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.291171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.291097    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwr9m\" (UniqueName: \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\") pod \"maas-api-revoke-sub-85237e45-swrcl\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.362169 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.362129    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:08.682756 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:12:08.682723    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e8af1_c978_43a0_9034_ebe162d75c87.slice/crio-4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b WatchSource:0}: Error finding container 4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b: Status 404 returned error can't find the container with id 4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b
Nov 03 06:12:08.683749 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:08.683721    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"]
Nov 03 06:12:09.106122 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:09.106078    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl" event={"ID":"1d0e8af1-c978-43a0-9034-ebe162d75c87","Type":"ContainerStarted","Data":"69c3294d791a7e98e1b51aebdf539257321d45e03296a24f785cc6a1c2cbe3a0"}
Nov 03 06:12:09.106638 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:09.106144    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl" event={"ID":"1d0e8af1-c978-43a0-9034-ebe162d75c87","Type":"ContainerStarted","Data":"4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b"}
Nov 03 06:12:09.152323 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:09.152257    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl" podStartSLOduration=2.152237514 podStartE2EDuration="2.152237514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:12:09.148708779 +0000 UTC m=+1017.560105859" watchObservedRunningTime="2026-11-03 06:12:09.152237514 +0000 UTC m=+1017.563634629"
Nov 03 06:12:09.749156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:09.749070    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8cdb804-9451-4a70-ad7f-224e18470bc9" path="/var/lib/kubelet/pods/d8cdb804-9451-4a70-ad7f-224e18470bc9/volumes"
Nov 03 06:12:10.042640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.042555    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"]
Nov 03 06:12:10.056962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.056925    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d26ce03-2mr6m"]
Nov 03 06:12:10.064410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.064375    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"]
Nov 03 06:12:10.069788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.069759    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8510b0c3-l9zh4"]
Nov 03 06:12:10.120361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.120324    4528 generic.go:356] "Generic (PLEG): container finished" podID="1d0e8af1-c978-43a0-9034-ebe162d75c87" containerID="69c3294d791a7e98e1b51aebdf539257321d45e03296a24f785cc6a1c2cbe3a0" exitCode=0
Nov 03 06:12:10.120882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:10.120408    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl" event={"ID":"1d0e8af1-c978-43a0-9034-ebe162d75c87","Type":"ContainerDied","Data":"69c3294d791a7e98e1b51aebdf539257321d45e03296a24f785cc6a1c2cbe3a0"}
Nov 03 06:12:11.292038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.292000    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:11.408662 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.408555    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\" (UniqueName: \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\") pod \"1d0e8af1-c978-43a0-9034-ebe162d75c87\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") "
Nov 03 06:12:11.408843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.408694    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\") pod \"1d0e8af1-c978-43a0-9034-ebe162d75c87\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") "
Nov 03 06:12:11.408843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.408807    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\") pod \"1d0e8af1-c978-43a0-9034-ebe162d75c87\" (UID: \"1d0e8af1-c978-43a0-9034-ebe162d75c87\") "
Nov 03 06:12:11.409252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.409203    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca" pod "1d0e8af1-c978-43a0-9034-ebe162d75c87" (UID: "1d0e8af1-c978-43a0-9034-ebe162d75c87"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:12:11.411234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.411181    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca" pod "1d0e8af1-c978-43a0-9034-ebe162d75c87" (UID: "1d0e8af1-c978-43a0-9034-ebe162d75c87"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:12:11.411461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.411433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m" pod "1d0e8af1-c978-43a0-9034-ebe162d75c87" (UID: "1d0e8af1-c978-43a0-9034-ebe162d75c87"). InnerVolumeSpecName "kube-api-access-zwr9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:12:11.510477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.510432    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:12:11.510477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.510475    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e8af1-c978-43a0-9034-ebe162d75c87-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:12:11.510754 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.510491    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwr9m\" (UniqueName: \"kubernetes.io/projected/1d0e8af1-c978-43a0-9034-ebe162d75c87-kube-api-access-zwr9m\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:12:11.757592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.757498    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07e8ca0e-7e80-4205-8a69-3b679fce4c3d" path="/var/lib/kubelet/pods/07e8ca0e-7e80-4205-8a69-3b679fce4c3d/volumes"
Nov 03 06:12:11.758050 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:11.758027    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="743e4996-6f32-412a-9d75-b0bab9479c7e" path="/var/lib/kubelet/pods/743e4996-6f32-412a-9d75-b0bab9479c7e/volumes"
Nov 03 06:12:12.045007 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:12.044920    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"]
Nov 03 06:12:12.052213 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:12.052153    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8180f860-486t8"]
Nov 03 06:12:12.148494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:12.148461    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"
Nov 03 06:12:12.148642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:12.148492    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl" event={"ID":"1d0e8af1-c978-43a0-9034-ebe162d75c87","Type":"ContainerDied","Data":"4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b"}
Nov 03 06:12:12.148642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:12.148566    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f91b0924ed0f5a5c89311e03ac62fb839453a3c828353bcd017d35a54530d7b"
Nov 03 06:12:13.031229 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:13.031185    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"]
Nov 03 06:12:13.042078 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:13.042045    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb043c0a-9mvdj"]
Nov 03 06:12:13.693232 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:12:13.693195    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:12:13.748367 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:13.748334    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08" path="/var/lib/kubelet/pods/7b0e5da2-a514-4c3d-961e-bb9a2d8c1d08/volumes"
Nov 03 06:12:13.748778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:13.748765    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e66745e-7b47-4051-b859-d04978db963a" path="/var/lib/kubelet/pods/9e66745e-7b47-4051-b859-d04978db963a/volumes"
Nov 03 06:12:22.031583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:22.031540    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"]
Nov 03 06:12:22.042850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:22.042816    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4920d59d-swvn6"]
Nov 03 06:12:23.748411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:23.748334    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e83b0d25-d8b3-419c-838d-dd93506f0910" path="/var/lib/kubelet/pods/e83b0d25-d8b3-419c-838d-dd93506f0910/volumes"
Nov 03 06:12:27.031457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:27.031419    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"]
Nov 03 06:12:27.042508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:27.042477    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c2f9f128-fwwhk"]
Nov 03 06:12:27.748649 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:27.748602    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77c33d08-25fe-4fd2-a4d3-7dee88dcda33" path="/var/lib/kubelet/pods/77c33d08-25fe-4fd2-a4d3-7dee88dcda33/volumes"
Nov 03 06:12:32.064312 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:32.064271    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"]
Nov 03 06:12:32.075366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:32.075338    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-de28d637-qdltd"]
Nov 03 06:12:33.747281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:33.747248    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82c35167-853e-43a2-b8c8-728174c0d3ab" path="/var/lib/kubelet/pods/82c35167-853e-43a2-b8c8-728174c0d3ab/volumes"
Nov 03 06:12:43.694603 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:12:43.694556    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:12:48.032150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:48.032113    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"]
Nov 03 06:12:48.043002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:48.042970    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbecaa07-prdpc"]
Nov 03 06:12:49.747415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:12:49.747382    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="257c6eaa-f097-4883-8619-523303809c4c" path="/var/lib/kubelet/pods/257c6eaa-f097-4883-8619-523303809c4c/volumes"
Nov 03 06:13:07.156288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.156250    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"]
Nov 03 06:13:07.157020 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.156999    4528 cpu_manager.go:414] "containerMap: removing container" podUID="1d0e8af1-c978-43a0-9034-ebe162d75c87" containerName="revoke-keys"
Nov 03 06:13:07.157125 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.157021    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0e8af1-c978-43a0-9034-ebe162d75c87" containerName="revoke-keys"
Nov 03 06:13:07.157190 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.157139    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d05fa7-9d57-46f6-a0ef-afea586291bd" containerName="istio-proxy"
Nov 03 06:13:07.157190 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.157152    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0e8af1-c978-43a0-9034-ebe162d75c87" containerName="revoke-keys"
Nov 03 06:13:07.160508 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.160488    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.174778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.174759    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:13:07.174897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.174805    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:13:07.186454 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.186430    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:13:07.186600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.186461    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:13:07.193284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.193254    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"]
Nov 03 06:13:07.206333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.206291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.206489 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.206364    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.206579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.206539    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.307987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.307948    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.308204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.308011    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.308204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.308057    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.308644 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.308620    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.310424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.310399    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.331144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.331119    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\") pod \"maas-api-revoke-sub-c1e271a9-tccwx\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.474680 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.474595    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:07.777103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:07.777074    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"]
Nov 03 06:13:07.777252 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:13:07.777231    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1ece506_39ed_45db_b463_4869fa461834.slice/crio-aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc WatchSource:0}: Error finding container aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc: Status 404 returned error can't find the container with id aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc
Nov 03 06:13:08.690308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:08.690271    4528 generic.go:356] "Generic (PLEG): container finished" podID="d1ece506-39ed-45db-b463-4869fa461834" containerID="91445f1a47fcd70790cccd328715799c8d7150bec92bd8b0af315139a5fb663a" exitCode=0
Nov 03 06:13:08.690692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:08.690351    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx" event={"ID":"d1ece506-39ed-45db-b463-4869fa461834","Type":"ContainerDied","Data":"91445f1a47fcd70790cccd328715799c8d7150bec92bd8b0af315139a5fb663a"}
Nov 03 06:13:08.690692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:08.690402    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx" event={"ID":"d1ece506-39ed-45db-b463-4869fa461834","Type":"ContainerStarted","Data":"aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc"}
Nov 03 06:13:09.833505 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.833477    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:09.937358 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.937321    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\") pod \"d1ece506-39ed-45db-b463-4869fa461834\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") "
Nov 03 06:13:09.937358 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.937360    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\") pod \"d1ece506-39ed-45db-b463-4869fa461834\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") "
Nov 03 06:13:09.937567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.937469    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\") pod \"d1ece506-39ed-45db-b463-4869fa461834\" (UID: \"d1ece506-39ed-45db-b463-4869fa461834\") "
Nov 03 06:13:09.937736 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.937709    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca" pod "d1ece506-39ed-45db-b463-4869fa461834" (UID: "d1ece506-39ed-45db-b463-4869fa461834"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:13:09.937847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.937831    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1ece506-39ed-45db-b463-4869fa461834-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:09.939565 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.939530    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj" pod "d1ece506-39ed-45db-b463-4869fa461834" (UID: "d1ece506-39ed-45db-b463-4869fa461834"). InnerVolumeSpecName "kube-api-access-ldblj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:13:09.939701 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:09.939599    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca" pod "d1ece506-39ed-45db-b463-4869fa461834" (UID: "d1ece506-39ed-45db-b463-4869fa461834"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:13:10.038820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:10.038713    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1ece506-39ed-45db-b463-4869fa461834-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:10.038820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:10.038758    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldblj\" (UniqueName: \"kubernetes.io/projected/d1ece506-39ed-45db-b463-4869fa461834-kube-api-access-ldblj\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:10.709438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:10.709408    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"
Nov 03 06:13:10.709640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:10.709444    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx" event={"ID":"d1ece506-39ed-45db-b463-4869fa461834","Type":"ContainerDied","Data":"aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc"}
Nov 03 06:13:10.709640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:10.709497    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadd8c1946dc62b871a9dcd82eda7ac25f8589b58a975c5906eb69e272fa79bc"
Nov 03 06:13:12.031015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:12.030980    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"]
Nov 03 06:13:12.042264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:12.042227    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f95078-czqdc"]
Nov 03 06:13:13.404114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.404077    4528 scope.go:122] "RemoveContainer" containerID="3ca58c491e59ccb4221967bc4a0d9f553a720a3ade6ac288c32b960cc05f16fe"
Nov 03 06:13:13.415443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.415422    4528 scope.go:122] "RemoveContainer" containerID="d17c77135d68d3ebf630574edffec6c273688fdcd6b6aa23c970a8bb9e44d0bd"
Nov 03 06:13:13.427960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.427926    4528 scope.go:122] "RemoveContainer" containerID="4d58850bdbe1a0aa05f3d0c6d0cd97182de9476dfd71104e61611211a4a6537a"
Nov 03 06:13:13.438558 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.438535    4528 scope.go:122] "RemoveContainer" containerID="42c76f5ef799bae37e8fb95b0e37c1fae636fb973c30a618cc9b9e2e33f8c2c1"
Nov 03 06:13:13.448418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.448397    4528 scope.go:122] "RemoveContainer" containerID="d4511a7589c490247d38100fc4951cf372178c9c7e81041d3b798efdf44b1fd9"
Nov 03 06:13:13.458474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.458451    4528 scope.go:122] "RemoveContainer" containerID="6a745d0c9d5ad90a4d85832d7ad8fc596f8a65152dcd2153269993e8b71cc0ae"
Nov 03 06:13:13.468811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.468769    4528 scope.go:122] "RemoveContainer" containerID="39370318a6458dcac71623af6ad0f20a6600faf9844807568feaedd796ec1665"
Nov 03 06:13:13.479191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.479168    4528 scope.go:122] "RemoveContainer" containerID="0b27d20c396a85871254611f379212a172721a6dd258a1416454973c1d359130"
Nov 03 06:13:13.489380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.489355    4528 scope.go:122] "RemoveContainer" containerID="6b74b2cb3c1fd5c2b46b34c6d905b5e9fcd3160aae19488d0ce78cea8a1fc997"
Nov 03 06:13:13.499507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.499483    4528 scope.go:122] "RemoveContainer" containerID="a114b5cdcbbc77e80c85328c2f19d0f5757f59a25485413fafce0e29ea8fe632"
Nov 03 06:13:13.509494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.509470    4528 scope.go:122] "RemoveContainer" containerID="52f85df3faa7a019439ac6d682a53d3d36c541eae95d01934f54be93bb0101d6"
Nov 03 06:13:13.519860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.519834    4528 scope.go:122] "RemoveContainer" containerID="e80f9ce3b041a37fdcf7f008586df23393670c1f67396234403334fa32645a08"
Nov 03 06:13:13.529843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.529817    4528 scope.go:122] "RemoveContainer" containerID="0ae8b18c1f658167e36624e99f86b62c01a46632513e968606d174573472cef6"
Nov 03 06:13:13.539554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.539535    4528 scope.go:122] "RemoveContainer" containerID="4fea7f61c8e07859e07ba966f6a9a0438ce3066f687a1ffc6e20f9e10e78627e"
Nov 03 06:13:13.549590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.549568    4528 scope.go:122] "RemoveContainer" containerID="f09449ee3a3fcc2cac0ef2b0c5915a69c61fc3e9dcd906b0c63ca320dfe46d2b"
Nov 03 06:13:13.560915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.560889    4528 scope.go:122] "RemoveContainer" containerID="ec9ca0ab412323025cb5fc55f7efb5b8a5758913926d7874036ed92a3986389b"
Nov 03 06:13:13.571133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.571113    4528 scope.go:122] "RemoveContainer" containerID="67946309c4d902bd1b814a555d2e32d48cd4163394e064d4bf10bf1410d15759"
Nov 03 06:13:13.696407 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:13:13.696375    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:13:13.747597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:13.747563    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c3d6cd9-858d-407f-ac70-f39bf95f488e" path="/var/lib/kubelet/pods/8c3d6cd9-858d-407f-ac70-f39bf95f488e/volumes"
Nov 03 06:13:26.056817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:26.054821    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"]
Nov 03 06:13:26.074324 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:26.074280    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9ef85531-s22tk"]
Nov 03 06:13:27.748015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:27.747982    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3ba9efb-1a22-45f8-b747-207be6f10fbe" path="/var/lib/kubelet/pods/a3ba9efb-1a22-45f8-b747-207be6f10fbe/volumes"
Nov 03 06:13:30.443839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.443806    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"]
Nov 03 06:13:30.444307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.444293    4528 cpu_manager.go:414] "containerMap: removing container" podUID="d1ece506-39ed-45db-b463-4869fa461834" containerName="revoke-keys"
Nov 03 06:13:30.444356 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.444308    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1ece506-39ed-45db-b463-4869fa461834" containerName="revoke-keys"
Nov 03 06:13:30.444412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.444400    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1ece506-39ed-45db-b463-4869fa461834" containerName="revoke-keys"
Nov 03 06:13:30.447835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.447816    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.466481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.466453    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"]
Nov 03 06:13:30.472391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.472366    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:13:30.472391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.472361    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:13:30.511887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.511854    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:13:30.512206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.512184    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:13:30.601993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.601953    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.602142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.602024    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.602142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.602123    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx6h5\" (UniqueName: \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.702867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.702744    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.702867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.702831    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.703102 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.702907    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx6h5\" (UniqueName: \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.703570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.703543    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.705207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.705185    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.726180 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.726156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx6h5\" (UniqueName: \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\") pod \"maas-api-revoke-sub-2e1f5b5f-d6lpl\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:30.761149 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:30.761124    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:31.067213 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:13:31.067178    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b759b3_e1c9_4313_8d53_0d263faa57ef.slice/crio-dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087 WatchSource:0}: Error finding container dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087: Status 404 returned error can't find the container with id dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087
Nov 03 06:13:31.068350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:31.068321    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"]
Nov 03 06:13:31.935474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:31.935436    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl" event={"ID":"07b759b3-e1c9-4313-8d53-0d263faa57ef","Type":"ContainerStarted","Data":"8e944eba9cf7dd790d6bd43a123e0e2276e695c58d399ec29f79ace9fe0548ce"}
Nov 03 06:13:31.935894 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:31.935503    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl" event={"ID":"07b759b3-e1c9-4313-8d53-0d263faa57ef","Type":"ContainerStarted","Data":"dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087"}
Nov 03 06:13:31.976730 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:31.976677    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl" podStartSLOduration=1.976664225 podStartE2EDuration="1.976664225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:13:31.973980181 +0000 UTC m=+1100.385377262" watchObservedRunningTime="2026-11-03 06:13:31.976664225 +0000 UTC m=+1100.388061344"
Nov 03 06:13:32.053857 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:13:32.053818    4528 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b759b3_e1c9_4313_8d53_0d263faa57ef.slice/crio-conmon-8e944eba9cf7dd790d6bd43a123e0e2276e695c58d399ec29f79ace9fe0548ce.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 06:13:32.943941 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:32.943911    4528 generic.go:356] "Generic (PLEG): container finished" podID="07b759b3-e1c9-4313-8d53-0d263faa57ef" containerID="8e944eba9cf7dd790d6bd43a123e0e2276e695c58d399ec29f79ace9fe0548ce" exitCode=0
Nov 03 06:13:32.944407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:32.943999    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl" event={"ID":"07b759b3-e1c9-4313-8d53-0d263faa57ef","Type":"ContainerDied","Data":"8e944eba9cf7dd790d6bd43a123e0e2276e695c58d399ec29f79ace9fe0548ce"}
Nov 03 06:13:34.094832 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.094807    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:34.239082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.239000    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\") pod \"07b759b3-e1c9-4313-8d53-0d263faa57ef\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") "
Nov 03 06:13:34.239221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.239113    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\") pod \"07b759b3-e1c9-4313-8d53-0d263faa57ef\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") "
Nov 03 06:13:34.239221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.239148    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\" (UniqueName: \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\") pod \"07b759b3-e1c9-4313-8d53-0d263faa57ef\" (UID: \"07b759b3-e1c9-4313-8d53-0d263faa57ef\") "
Nov 03 06:13:34.239454 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.239426    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca" pod "07b759b3-e1c9-4313-8d53-0d263faa57ef" (UID: "07b759b3-e1c9-4313-8d53-0d263faa57ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:13:34.241260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.241234    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca" pod "07b759b3-e1c9-4313-8d53-0d263faa57ef" (UID: "07b759b3-e1c9-4313-8d53-0d263faa57ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:13:34.241340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.241279    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5" pod "07b759b3-e1c9-4313-8d53-0d263faa57ef" (UID: "07b759b3-e1c9-4313-8d53-0d263faa57ef"). InnerVolumeSpecName "kube-api-access-lx6h5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:13:34.340612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.340574    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:34.340612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.340605    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx6h5\" (UniqueName: \"kubernetes.io/projected/07b759b3-e1c9-4313-8d53-0d263faa57ef-kube-api-access-lx6h5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:34.340612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.340614    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07b759b3-e1c9-4313-8d53-0d263faa57ef-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:34.959697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.959668    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"
Nov 03 06:13:34.959918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.959708    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl" event={"ID":"07b759b3-e1c9-4313-8d53-0d263faa57ef","Type":"ContainerDied","Data":"dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087"}
Nov 03 06:13:34.959918 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:34.959754    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd0390c2a354e8855bf13cbb13fa859e05e1fc259888c717b69daa32a4d90087"
Nov 03 06:13:41.770469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.770433    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"]
Nov 03 06:13:41.771158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.771140    4528 cpu_manager.go:414] "containerMap: removing container" podUID="07b759b3-e1c9-4313-8d53-0d263faa57ef" containerName="revoke-keys"
Nov 03 06:13:41.771158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.771159    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b759b3-e1c9-4313-8d53-0d263faa57ef" containerName="revoke-keys"
Nov 03 06:13:41.771294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.771279    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b759b3-e1c9-4313-8d53-0d263faa57ef" containerName="revoke-keys"
Nov 03 06:13:41.775584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.775559    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.793005 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.792978    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"]
Nov 03 06:13:41.797724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.797700    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:13:41.797724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.797701    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:13:41.800855 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.800835    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:13:41.800855 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.800846    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:13:41.809136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.809112    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.809250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.809159    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.809250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.809186    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d4q4\" (UniqueName: \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.909853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.909820    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.910039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.909876    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.910039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.909902    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d4q4\" (UniqueName: \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.910592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.910564    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.912117 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.912100    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:41.933183 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:41.933155    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d4q4\" (UniqueName: \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\") pod \"maas-api-revoke-sub-ec843018-mwdpd\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:42.092354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:42.092321    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:42.450514 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:13:42.450482    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc342fcbd_bae7_4776_9e79_41e49b647daa.slice/crio-e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb WatchSource:0}: Error finding container e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb: Status 404 returned error can't find the container with id e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb
Nov 03 06:13:42.451625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:42.451595    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"]
Nov 03 06:13:43.024015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:43.023918    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd" event={"ID":"c342fcbd-bae7-4776-9e79-41e49b647daa","Type":"ContainerStarted","Data":"5d575dfea9aad1e47604d5fcc42d244064dccaa317777e3f1730d39b5a7ccfa4"}
Nov 03 06:13:43.024015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:43.023991    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd" event={"ID":"c342fcbd-bae7-4776-9e79-41e49b647daa","Type":"ContainerStarted","Data":"e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb"}
Nov 03 06:13:43.063355 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:43.063297    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd" podStartSLOduration=2.063280946 podStartE2EDuration="2.063280946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:13:43.062267311 +0000 UTC m=+1111.473664389" watchObservedRunningTime="2026-11-03 06:13:43.063280946 +0000 UTC m=+1111.474678024"
Nov 03 06:13:43.694247 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:13:43.694166    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:13:44.034803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:44.034695    4528 generic.go:356] "Generic (PLEG): container finished" podID="c342fcbd-bae7-4776-9e79-41e49b647daa" containerID="5d575dfea9aad1e47604d5fcc42d244064dccaa317777e3f1730d39b5a7ccfa4" exitCode=0
Nov 03 06:13:44.035217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:44.034813    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd" event={"ID":"c342fcbd-bae7-4776-9e79-41e49b647daa","Type":"ContainerDied","Data":"5d575dfea9aad1e47604d5fcc42d244064dccaa317777e3f1730d39b5a7ccfa4"}
Nov 03 06:13:45.269579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.269548    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:45.339270 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.339230    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\" (UniqueName: \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\") pod \"c342fcbd-bae7-4776-9e79-41e49b647daa\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") "
Nov 03 06:13:45.339488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.339365    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\") pod \"c342fcbd-bae7-4776-9e79-41e49b647daa\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") "
Nov 03 06:13:45.339488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.339436    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\") pod \"c342fcbd-bae7-4776-9e79-41e49b647daa\" (UID: \"c342fcbd-bae7-4776-9e79-41e49b647daa\") "
Nov 03 06:13:45.340134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.340085    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca" pod "c342fcbd-bae7-4776-9e79-41e49b647daa" (UID: "c342fcbd-bae7-4776-9e79-41e49b647daa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:13:45.342552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.342529    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca" pod "c342fcbd-bae7-4776-9e79-41e49b647daa" (UID: "c342fcbd-bae7-4776-9e79-41e49b647daa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:13:45.342837 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.342783    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4" pod "c342fcbd-bae7-4776-9e79-41e49b647daa" (UID: "c342fcbd-bae7-4776-9e79-41e49b647daa"). InnerVolumeSpecName "kube-api-access-6d4q4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:13:45.442304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.441940    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:45.442304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.441983    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c342fcbd-bae7-4776-9e79-41e49b647daa-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:45.442304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:45.442000    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d4q4\" (UniqueName: \"kubernetes.io/projected/c342fcbd-bae7-4776-9e79-41e49b647daa-kube-api-access-6d4q4\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:13:46.094453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:46.094345    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd" event={"ID":"c342fcbd-bae7-4776-9e79-41e49b647daa","Type":"ContainerDied","Data":"e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb"}
Nov 03 06:13:46.094717 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:46.094690    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e70ece07a217d9c88b053fe2e9a3ee903780c298018bc1603ad34d832c8fcceb"
Nov 03 06:13:46.094947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:46.094932    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"
Nov 03 06:13:54.039502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:54.039409    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"]
Nov 03 06:13:54.050713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:54.050682    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f6a8d76-hmwhs"]
Nov 03 06:13:55.748925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:13:55.748887    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="898d8715-d64b-42d8-b573-b2b7f9ba9756" path="/var/lib/kubelet/pods/898d8715-d64b-42d8-b573-b2b7f9ba9756/volumes"
Nov 03 06:14:00.038420 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:00.038363    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"]
Nov 03 06:14:00.063208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:00.063175    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b2c43dd0-jbg8c"]
Nov 03 06:14:01.748568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:01.748533    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f391aab-116a-44f6-b17c-55ebe692c8d6" path="/var/lib/kubelet/pods/7f391aab-116a-44f6-b17c-55ebe692c8d6/volumes"
Nov 03 06:14:07.031321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:07.031277    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"]
Nov 03 06:14:07.042262 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:07.042231    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d834b2b-72hvq"]
Nov 03 06:14:07.749108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:07.749078    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3420df3c-9273-4979-9a34-42e329a21d1c" path="/var/lib/kubelet/pods/3420df3c-9273-4979-9a34-42e329a21d1c/volumes"
Nov 03 06:14:09.374927 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.374888    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"]
Nov 03 06:14:09.375554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.375533    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c342fcbd-bae7-4776-9e79-41e49b647daa" containerName="revoke-keys"
Nov 03 06:14:09.375642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.375555    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c342fcbd-bae7-4776-9e79-41e49b647daa" containerName="revoke-keys"
Nov 03 06:14:09.375727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.375713    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c342fcbd-bae7-4776-9e79-41e49b647daa" containerName="revoke-keys"
Nov 03 06:14:09.380187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.380164    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.400753 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.400725    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"]
Nov 03 06:14:09.407226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.407171    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:14:09.407369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.407235    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:14:09.407523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.407495    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:14:09.408865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.408845    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:14:09.516516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.516474    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.516687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.516530    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.516687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.516670    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gjxq\" (UniqueName: \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.617287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.617255    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gjxq\" (UniqueName: \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.617427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.617322    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.617427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.617350    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.617982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.617951    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.619646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.619621    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.640581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.640523    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gjxq\" (UniqueName: \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\") pod \"maas-api-revoke-sub-046e1343-xstgh\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.693665 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.693639    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:09.995244 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:14:09.995215    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd2cdbf_b452_4833_be9a_0c7c8e5dd8c3.slice/crio-ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e WatchSource:0}: Error finding container ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e: Status 404 returned error can't find the container with id ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e
Nov 03 06:14:09.996396 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:09.996372    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"]
Nov 03 06:14:10.362777 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:10.362719    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh" event={"ID":"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3","Type":"ContainerStarted","Data":"7985c8a43ae3086bf2bd8fb8773c7286d999e9e9c7a8afddb71673095637b2cc"}
Nov 03 06:14:10.362985 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:10.362879    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh" event={"ID":"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3","Type":"ContainerStarted","Data":"ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e"}
Nov 03 06:14:10.403001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:10.402942    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh" podStartSLOduration=1.402928162 podStartE2EDuration="1.402928162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:14:10.402241877 +0000 UTC m=+1138.813638956" watchObservedRunningTime="2026-11-03 06:14:10.402928162 +0000 UTC m=+1138.814325292"
Nov 03 06:14:11.372968 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:11.372929    4528 generic.go:356] "Generic (PLEG): container finished" podID="bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" containerID="7985c8a43ae3086bf2bd8fb8773c7286d999e9e9c7a8afddb71673095637b2cc" exitCode=0
Nov 03 06:14:11.373144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:11.372993    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh" event={"ID":"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3","Type":"ContainerDied","Data":"7985c8a43ae3086bf2bd8fb8773c7286d999e9e9c7a8afddb71673095637b2cc"}
Nov 03 06:14:12.514884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.514857    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:12.645847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.645710    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\" (UniqueName: \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\") pod \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") "
Nov 03 06:14:12.645847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.645840    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\") pod \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") "
Nov 03 06:14:12.646105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.645927    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\") pod \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\" (UID: \"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3\") "
Nov 03 06:14:12.646308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.646284    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca" pod "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" (UID: "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:14:12.648047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.647952    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca" pod "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" (UID: "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:14:12.648166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.648143    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq" pod "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" (UID: "bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3"). InnerVolumeSpecName "kube-api-access-7gjxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:14:12.747617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.747584    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:12.747617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.747620    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gjxq\" (UniqueName: \"kubernetes.io/projected/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-kube-api-access-7gjxq\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:12.747875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:12.747639    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:13.390815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.390766    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"
Nov 03 06:14:13.390815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.390808    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh" event={"ID":"bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3","Type":"ContainerDied","Data":"ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e"}
Nov 03 06:14:13.391113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.390852    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6c63c68c5ff5003f764647409455c9796541836170144e171e21643fbcdc2e"
Nov 03 06:14:13.693339 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:14:13.693254    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:14:13.698415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.698392    4528 scope.go:122] "RemoveContainer" containerID="4f60f0e7656e3c0d702183e4474a00c8fbb78d4208c9a5a19f99d6214bbe360a"
Nov 03 06:14:13.708726 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.708706    4528 scope.go:122] "RemoveContainer" containerID="fce25e2c7f6f170f3f216a9471c7a3b99f77d9b5856725b47852efb5087c270c"
Nov 03 06:14:13.718227 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.718208    4528 scope.go:122] "RemoveContainer" containerID="57849d59993c7623094a6a149a3ff9bdd1ff17d42471de55ac8d21b056a7f94f"
Nov 03 06:14:13.727274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.727253    4528 scope.go:122] "RemoveContainer" containerID="d413667292ade9a06fd8028e40f0cca07abb12e08c270939c4e4de6d059b1630"
Nov 03 06:14:13.738475 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.738454    4528 scope.go:122] "RemoveContainer" containerID="1f8a94f7ee22fa2a917dc0f91ae189f615bb7001cd46aa8fb54bdadb486a7a1f"
Nov 03 06:14:13.749184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:13.749166    4528 scope.go:122] "RemoveContainer" containerID="bd9232f5e4ec9400b5eebabdb4ff20fbbb0ee3c66d142472789f3e3868ad5be9"
Nov 03 06:14:19.738345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.738306    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"]
Nov 03 06:14:19.738851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.738782    4528 cpu_manager.go:414] "containerMap: removing container" podUID="bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" containerName="revoke-keys"
Nov 03 06:14:19.738851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.738812    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" containerName="revoke-keys"
Nov 03 06:14:19.738974 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.738911    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" containerName="revoke-keys"
Nov 03 06:14:19.744252 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.744219    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:19.759255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.759227    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"]
Nov 03 06:14:19.764451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.764423    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:14:19.764574 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.764467    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:14:19.764574 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.764534    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:14:19.764671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.764604    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:14:19.918382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.918331    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b96n\" (UniqueName: \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:19.918573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.918393    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:19.918573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:19.918463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.019300 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.019196    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b96n\" (UniqueName: \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.019300 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.019257    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.019534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.019310    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.020035 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.020002    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.021716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.021682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.042168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.042141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b96n\" (UniqueName: \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\") pod \"maas-api-revoke-sub-d3fba056-qw4cc\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.067216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.067189    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:20.371854 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:14:20.371779    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4406891_25a7_4b89_bfd6_d9823855221a.slice/crio-72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c WatchSource:0}: Error finding container 72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c: Status 404 returned error can't find the container with id 72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c
Nov 03 06:14:20.373977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.373947    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"]
Nov 03 06:14:20.450021 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:20.449975    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc" event={"ID":"e4406891-25a7-4b89-bfd6-d9823855221a","Type":"ContainerStarted","Data":"72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c"}
Nov 03 06:14:21.037186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:21.037109    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"]
Nov 03 06:14:21.051742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:21.051703    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70051005-z98sh"]
Nov 03 06:14:21.465095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:21.465050    4528 generic.go:356] "Generic (PLEG): container finished" podID="e4406891-25a7-4b89-bfd6-d9823855221a" containerID="27ee58a7506ac6872464d1d76ccba2b77b8612ee309b74ccc7e4f0b5cf17d495" exitCode=0
Nov 03 06:14:21.465348 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:21.465324    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc" event={"ID":"e4406891-25a7-4b89-bfd6-d9823855221a","Type":"ContainerDied","Data":"27ee58a7506ac6872464d1d76ccba2b77b8612ee309b74ccc7e4f0b5cf17d495"}
Nov 03 06:14:21.759876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:21.759749    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e000c21d-1a21-423f-b288-fddb0c903536" path="/var/lib/kubelet/pods/e000c21d-1a21-423f-b288-fddb0c903536/volumes"
Nov 03 06:14:23.471026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.470990    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:23.650871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.650241    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc" event={"ID":"e4406891-25a7-4b89-bfd6-d9823855221a","Type":"ContainerDied","Data":"72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c"}
Nov 03 06:14:23.650871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.650305    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72842ee60056959a085fff6dc72f16bf4b3179419f64bcb0a82ab0882761c99c"
Nov 03 06:14:23.650871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.650376    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"
Nov 03 06:14:23.666815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.659422    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\" (UniqueName: \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\") pod \"e4406891-25a7-4b89-bfd6-d9823855221a\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") "
Nov 03 06:14:23.666815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.659504    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\") pod \"e4406891-25a7-4b89-bfd6-d9823855221a\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") "
Nov 03 06:14:23.666815 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.659637    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\") pod \"e4406891-25a7-4b89-bfd6-d9823855221a\" (UID: \"e4406891-25a7-4b89-bfd6-d9823855221a\") "
Nov 03 06:14:23.686029 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.685990    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca" pod "e4406891-25a7-4b89-bfd6-d9823855221a" (UID: "e4406891-25a7-4b89-bfd6-d9823855221a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:14:23.737283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.737244    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca" pod "e4406891-25a7-4b89-bfd6-d9823855221a" (UID: "e4406891-25a7-4b89-bfd6-d9823855221a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:14:23.743048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.743014    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n" pod "e4406891-25a7-4b89-bfd6-d9823855221a" (UID: "e4406891-25a7-4b89-bfd6-d9823855221a"). InnerVolumeSpecName "kube-api-access-8b96n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:14:23.766337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.766305    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:23.767488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.767465    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b96n\" (UniqueName: \"kubernetes.io/projected/e4406891-25a7-4b89-bfd6-d9823855221a-kube-api-access-8b96n\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:23.767627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:23.767613    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4406891-25a7-4b89-bfd6-d9823855221a-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:26.030026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:26.029990    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"]
Nov 03 06:14:26.040720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:26.040689    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-622f06ef-7bksc"]
Nov 03 06:14:27.029260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:27.029221    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"]
Nov 03 06:14:27.040561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:27.040528    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-183fae31-x2n87"]
Nov 03 06:14:27.747308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:27.747277    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1597ac30-419a-47f2-8802-15fa12ac07f9" path="/var/lib/kubelet/pods/1597ac30-419a-47f2-8802-15fa12ac07f9/volumes"
Nov 03 06:14:27.747703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:27.747690    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4591b4f2-23ed-4282-abdf-ac5e65fe0b2b" path="/var/lib/kubelet/pods/4591b4f2-23ed-4282-abdf-ac5e65fe0b2b/volumes"
Nov 03 06:14:33.039445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:33.039400    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"]
Nov 03 06:14:33.045743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:33.045712    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-440f6461-sbprc"]
Nov 03 06:14:33.747591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:33.747554    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb112c3f-26a0-460f-b06c-608a1764e49b" path="/var/lib/kubelet/pods/bb112c3f-26a0-460f-b06c-608a1764e49b/volumes"
Nov 03 06:14:38.793835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:38.793448    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-8mbmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:14:38.793835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:38.793515    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-8mbmr" podUID="11852422-e9b2-4cc4-a315-644dd38888f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:14:43.028923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:43.028876    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"]
Nov 03 06:14:43.039899 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:43.039866    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b088b577-zdzdn"]
Nov 03 06:14:43.693727 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:14:43.693683    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:14:43.748144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:43.748111    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dbd1c94-a96b-4420-bb10-5bede5215c7d" path="/var/lib/kubelet/pods/3dbd1c94-a96b-4420-bb10-5bede5215c7d/volumes"
Nov 03 06:14:44.453080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.453043    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"]
Nov 03 06:14:44.453735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.453713    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e4406891-25a7-4b89-bfd6-d9823855221a" containerName="revoke-keys"
Nov 03 06:14:44.453863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.453737    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4406891-25a7-4b89-bfd6-d9823855221a" containerName="revoke-keys"
Nov 03 06:14:44.453937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.453913    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4406891-25a7-4b89-bfd6-d9823855221a" containerName="revoke-keys"
Nov 03 06:14:44.458349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.458328    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.475888 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.475858    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"]
Nov 03 06:14:44.484633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.482182    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:14:44.484633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.482200    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:14:44.484633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.482312    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:14:44.485140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.485122    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:14:44.489758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.489736    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.489879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.489776    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.489925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.489911    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrlt5\" (UniqueName: \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.590398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.590368    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.590619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.590414    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.590619 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.590487    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrlt5\" (UniqueName: \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.590995 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.590976    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.592833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.592812    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.613412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.613382    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrlt5\" (UniqueName: \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\") pod \"maas-api-revoke-sub-ae594979-8vvxn\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:44.773056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:44.772969    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:45.075685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:45.075652    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"]
Nov 03 06:14:45.076130 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:14:45.076103    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65ebe2eb_31e8_4b42_b39d_7f6c7de1eaa5.slice/crio-b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d WatchSource:0}: Error finding container b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d: Status 404 returned error can't find the container with id b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d
Nov 03 06:14:45.896744 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:45.896705    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn" event={"ID":"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5","Type":"ContainerStarted","Data":"3db382fdcd354876a68c9ea802f7d57970eb4c6817aa96294041bd02dd0b6bf0"}
Nov 03 06:14:45.897163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:45.896760    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn" event={"ID":"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5","Type":"ContainerStarted","Data":"b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d"}
Nov 03 06:14:45.939887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:45.939838    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn" podStartSLOduration=1.939824228 podStartE2EDuration="1.939824228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:14:45.935740675 +0000 UTC m=+1174.347137754" watchObservedRunningTime="2026-11-03 06:14:45.939824228 +0000 UTC m=+1174.351221303"
Nov 03 06:14:46.914651 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:46.914613    4528 generic.go:356] "Generic (PLEG): container finished" podID="65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" containerID="3db382fdcd354876a68c9ea802f7d57970eb4c6817aa96294041bd02dd0b6bf0" exitCode=0
Nov 03 06:14:46.915145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:46.914672    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn" event={"ID":"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5","Type":"ContainerDied","Data":"3db382fdcd354876a68c9ea802f7d57970eb4c6817aa96294041bd02dd0b6bf0"}
Nov 03 06:14:48.064471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.064446    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:48.124170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.124133    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\") pod \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") "
Nov 03 06:14:48.124330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.124260    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\") pod \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") "
Nov 03 06:14:48.124330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.124301    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\" (UniqueName: \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\") pod \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\" (UID: \"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5\") "
Nov 03 06:14:48.124698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.124665    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca" pod "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" (UID: "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:14:48.126288 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.126262    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca" pod "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" (UID: "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:14:48.126624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.126599    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5" pod "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" (UID: "65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5"). InnerVolumeSpecName "kube-api-access-wrlt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:14:48.226081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.225999    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:48.226081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.226029    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrlt5\" (UniqueName: \"kubernetes.io/projected/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-kube-api-access-wrlt5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:48.226081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.226038    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:14:48.931060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.931019    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn" event={"ID":"65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5","Type":"ContainerDied","Data":"b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d"}
Nov 03 06:14:48.931060 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.931040    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"
Nov 03 06:14:48.931274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:48.931074    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b09bc3c6b759ac43982338f717a55306eb83914796c3c1a87006e7af4bf5b59d"
Nov 03 06:14:55.032164 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:55.032119    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"]
Nov 03 06:14:55.043626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:55.043594    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-932625f0-2l2nc"]
Nov 03 06:14:55.747582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:55.747546    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="038fb417-ce3c-4c44-85ea-cdd567b8d94a" path="/var/lib/kubelet/pods/038fb417-ce3c-4c44-85ea-cdd567b8d94a/volumes"
Nov 03 06:14:56.029236 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:56.029149    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"]
Nov 03 06:14:56.040756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:56.040723    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d635ee9-6bfzr"]
Nov 03 06:14:57.754100 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:57.754067    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bb1afc-8271-488d-97b4-8360da80568d" path="/var/lib/kubelet/pods/03bb1afc-8271-488d-97b4-8360da80568d/volumes"
Nov 03 06:14:58.037137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:58.037048    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"]
Nov 03 06:14:58.048130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:58.048099    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e007df9-c5mmk"]
Nov 03 06:14:59.040477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.040433    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"]
Nov 03 06:14:59.052408 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.052370    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"]
Nov 03 06:14:59.058888 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.058857    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-93228110-cc6g6"]
Nov 03 06:14:59.064807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.064744    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24e53cd5-58m7h"]
Nov 03 06:14:59.780445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.780408    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651f869b-7616-4abf-8e56-3ef001213926" path="/var/lib/kubelet/pods/651f869b-7616-4abf-8e56-3ef001213926/volumes"
Nov 03 06:14:59.781372 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.781348    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766bd37a-3a4a-4b45-bc41-2ca774589377" path="/var/lib/kubelet/pods/766bd37a-3a4a-4b45-bc41-2ca774589377/volumes"
Nov 03 06:14:59.782095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:14:59.782075    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c048cd5e-c213-41df-8e2d-d5ee8ff17e5e" path="/var/lib/kubelet/pods/c048cd5e-c213-41df-8e2d-d5ee8ff17e5e/volumes"
Nov 03 06:15:00.172329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.172285    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:00.173677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.173633    4528 cpu_manager.go:414] "containerMap: removing container" podUID="65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" containerName="revoke-keys"
Nov 03 06:15:00.176077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.176054    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" containerName="revoke-keys"
Nov 03 06:15:00.176423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.176404    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" containerName="revoke-keys"
Nov 03 06:15:00.209043 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.209003    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:00.209395 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.209373    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.254382 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.235340    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:15:00.254629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.249562    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:15:00.254750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.249741    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:15:00.286038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.285715    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.286038 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.285842    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7xng\" (UniqueName: \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.389818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.389409    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.389818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.389556    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7xng\" (UniqueName: \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.390822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.390772    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.425911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.425832    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7xng\" (UniqueName: \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\") pod \"maas-api-key-cleanup-29894775-stfg6\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:00.612350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:00.611988    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:01.672319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:01.672262    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:02.228915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:02.228881    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerStarted","Data":"ded48df7b9f3fb8bf6860d2c98ba9c1010c1abf7e1c51ebb608b12cd2b9708c9"}
Nov 03 06:15:02.229110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:02.228938    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerStarted","Data":"89e3784f20e34c1028bee2b68f7edd0668f24f1b6a25c7cb1ca9072bb3d50e77"}
Nov 03 06:15:02.271392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:02.271338    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" podStartSLOduration=2.271322456 podStartE2EDuration="2.271322456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:15:02.2680721 +0000 UTC m=+1190.679469180" watchObservedRunningTime="2026-11-03 06:15:02.271322456 +0000 UTC m=+1190.682719569"
Nov 03 06:15:03.247816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:03.247755    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerID="ded48df7b9f3fb8bf6860d2c98ba9c1010c1abf7e1c51ebb608b12cd2b9708c9" exitCode=60
Nov 03 06:15:03.248222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:03.247845    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerDied","Data":"ded48df7b9f3fb8bf6860d2c98ba9c1010c1abf7e1c51ebb608b12cd2b9708c9"}
Nov 03 06:15:03.248222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:03.248141    4528 scope.go:122] "RemoveContainer" containerID="ded48df7b9f3fb8bf6860d2c98ba9c1010c1abf7e1c51ebb608b12cd2b9708c9"
Nov 03 06:15:04.257944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:04.257905    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerID="afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc" exitCode=60
Nov 03 06:15:04.258421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:04.257991    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerDied","Data":"afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc"}
Nov 03 06:15:04.258421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:04.258054    4528 scope.go:122] "RemoveContainer" containerID="ded48df7b9f3fb8bf6860d2c98ba9c1010c1abf7e1c51ebb608b12cd2b9708c9"
Nov 03 06:15:04.258421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:04.258335    4528 scope.go:122] "RemoveContainer" containerID="afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc"
Nov 03 06:15:04.258608 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:15:04.258589    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894775-stfg6_odh-ai-gateway-infra(e1eeff51-f66a-4cdc-b677-4175d49b3a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e"
Nov 03 06:15:05.267314 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:05.267287    4528 scope.go:122] "RemoveContainer" containerID="afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc"
Nov 03 06:15:05.267731 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:15:05.267541    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894775-stfg6_odh-ai-gateway-infra(e1eeff51-f66a-4cdc-b677-4175d49b3a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e"
Nov 03 06:15:07.028116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:07.028077    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"]
Nov 03 06:15:07.039430 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:07.039401    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-454693e6-czm7s"]
Nov 03 06:15:07.747397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:07.747363    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d4b7e6b-4199-4928-a724-6064c2e1aa0a" path="/var/lib/kubelet/pods/5d4b7e6b-4199-4928-a724-6064c2e1aa0a/volumes"
Nov 03 06:15:08.032841 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:08.032726    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"]
Nov 03 06:15:08.040865 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:08.040834    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b2757ba-w6fhk"]
Nov 03 06:15:09.747524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:09.747488    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0e08e38-2bfd-4099-a13c-9d85807c9c2b" path="/var/lib/kubelet/pods/c0e08e38-2bfd-4099-a13c-9d85807c9c2b/volumes"
Nov 03 06:15:13.028926 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.028886    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"]
Nov 03 06:15:13.040426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.040398    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-afbdb8f0-kxsp7"]
Nov 03 06:15:13.691424 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:15:13.691387    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:15:13.747461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.747426    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c6dd176-5325-4ac3-b302-8c2cc615e7c6" path="/var/lib/kubelet/pods/9c6dd176-5325-4ac3-b302-8c2cc615e7c6/volumes"
Nov 03 06:15:13.800899 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.800869    4528 scope.go:122] "RemoveContainer" containerID="c63650f08bad0358da32b86277e52911aa4f55758fe0d824022d3f436f67bc30"
Nov 03 06:15:13.813409 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.813383    4528 scope.go:122] "RemoveContainer" containerID="62980d1a48298286e294f2d71078b2b0822b5a35cc7324f40c701b8e951e57fe"
Nov 03 06:15:13.822443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.822425    4528 scope.go:122] "RemoveContainer" containerID="da2c57d13ecae4815902de36e6ae943c9d932481676c813f9b8ae998b5bc47a3"
Nov 03 06:15:13.831628 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.831600    4528 scope.go:122] "RemoveContainer" containerID="a02484401a183e003229fa57920333a5b689d699d8b8e31b769c5ad9afb2487c"
Nov 03 06:15:13.841357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:13.841336    4528 scope.go:122] "RemoveContainer" containerID="a1826aa26b6febd799163afe76cffbc071da9c30ccf007b21946c7f1ceb53101"
Nov 03 06:15:17.742358 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:17.742325    4528 scope.go:122] "RemoveContainer" containerID="afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc"
Nov 03 06:15:18.368663 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:18.368618    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerStarted","Data":"919631454fb065b8453194ab9e0b51ff3861bed05e2f3b867f38137ce77df55f"}
Nov 03 06:15:19.378067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:19.378029    4528 generic.go:356] "Generic (PLEG): container finished" podID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerID="919631454fb065b8453194ab9e0b51ff3861bed05e2f3b867f38137ce77df55f" exitCode=60
Nov 03 06:15:19.378512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:19.378104    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerDied","Data":"919631454fb065b8453194ab9e0b51ff3861bed05e2f3b867f38137ce77df55f"}
Nov 03 06:15:19.378512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:19.378186    4528 scope.go:122] "RemoveContainer" containerID="afb1b602ed3539f1d93f75f750973d4b79623f43f82e46cc6bb43041f0c5f0fc"
Nov 03 06:15:19.378637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:19.378522    4528 scope.go:122] "RemoveContainer" containerID="919631454fb065b8453194ab9e0b51ff3861bed05e2f3b867f38137ce77df55f"
Nov 03 06:15:19.378838 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:15:19.378813    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894775-stfg6_odh-ai-gateway-infra(e1eeff51-f66a-4cdc-b677-4175d49b3a5e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e"
Nov 03 06:15:19.420929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:19.420894    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:20.557052 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.557024    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:20.698423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.698341    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\") pod \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") "
Nov 03 06:15:20.698423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.698407    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\" (UniqueName: \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\") pod \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\" (UID: \"e1eeff51-f66a-4cdc-b677-4175d49b3a5e\") "
Nov 03 06:15:20.698704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.698679    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca" pod "e1eeff51-f66a-4cdc-b677-4175d49b3a5e" (UID: "e1eeff51-f66a-4cdc-b677-4175d49b3a5e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:15:20.700403 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.700378    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng" pod "e1eeff51-f66a-4cdc-b677-4175d49b3a5e" (UID: "e1eeff51-f66a-4cdc-b677-4175d49b3a5e"). InnerVolumeSpecName "kube-api-access-g7xng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:15:20.799958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.799924    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-maas-api-service-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:20.799958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:20.799951    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7xng\" (UniqueName: \"kubernetes.io/projected/e1eeff51-f66a-4cdc-b677-4175d49b3a5e-kube-api-access-g7xng\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:21.414601 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.414564    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"
Nov 03 06:15:21.414807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.414606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6" event={"ID":"e1eeff51-f66a-4cdc-b677-4175d49b3a5e","Type":"ContainerDied","Data":"89e3784f20e34c1028bee2b68f7edd0668f24f1b6a25c7cb1ca9072bb3d50e77"}
Nov 03 06:15:21.414807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.414668    4528 scope.go:122] "RemoveContainer" containerID="919631454fb065b8453194ab9e0b51ff3861bed05e2f3b867f38137ce77df55f"
Nov 03 06:15:21.467704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.467672    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:21.478872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.478845    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894775-stfg6"]
Nov 03 06:15:21.748741 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:21.748665    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" path="/var/lib/kubelet/pods/e1eeff51-f66a-4cdc-b677-4175d49b3a5e/volumes"
Nov 03 06:15:29.030739 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.030638    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"]
Nov 03 06:15:29.042206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.042180    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b29e034a-6sfg4"]
Nov 03 06:15:29.204786 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.204744    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"]
Nov 03 06:15:29.205294 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205274    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.205397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205296    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.205397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205318    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.205397 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205325    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.205549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205450    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.205549 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.205465    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:29.210156 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.210133    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.225778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.225742    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:15:29.226097 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.226071    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:15:29.226202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.226102    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:15:29.226202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.226150    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:15:29.228030 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.228010    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"]
Nov 03 06:15:29.229529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.229504    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.229635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.229614    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vq52\" (UniqueName: \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.229697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.229681    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.330593 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.330559    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vq52\" (UniqueName: \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.330779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.330616    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.330779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.330673    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.331333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.331310    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.333031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.333007    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.353177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.353153    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vq52\" (UniqueName: \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\") pod \"maas-api-revoke-sub-79805ca9-xxjhf\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.523271 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.523235    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:29.748597 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.748523    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e1b35a-af1c-49da-b330-3ee39a7ad4a2" path="/var/lib/kubelet/pods/e2e1b35a-af1c-49da-b330-3ee39a7ad4a2/volumes"
Nov 03 06:15:29.824426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:29.824390    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"]
Nov 03 06:15:29.825167 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:15:29.825143    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcc6b768_fd1a_4830_b8db_eaa24e294102.slice/crio-95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25 WatchSource:0}: Error finding container 95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25: Status 404 returned error can't find the container with id 95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25
Nov 03 06:15:30.538094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:30.538012    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf" event={"ID":"dcc6b768-fd1a-4830-b8db-eaa24e294102","Type":"ContainerStarted","Data":"15167bff4564fc610295e4134629dec685a596ea5e50237651873a2ac1cfa06f"}
Nov 03 06:15:30.538094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:30.538066    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf" event={"ID":"dcc6b768-fd1a-4830-b8db-eaa24e294102","Type":"ContainerStarted","Data":"95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25"}
Nov 03 06:15:30.579958 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:30.579901    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf" podStartSLOduration=1.579884813 podStartE2EDuration="1.579884813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:15:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:15:30.577334456 +0000 UTC m=+1218.988731535" watchObservedRunningTime="2026-11-03 06:15:30.579884813 +0000 UTC m=+1218.991281890"
Nov 03 06:15:31.027896 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:31.027863    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"]
Nov 03 06:15:31.039155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:31.039124    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f23d1a4f-gk2hp"]
Nov 03 06:15:31.547410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:31.547375    4528 generic.go:356] "Generic (PLEG): container finished" podID="dcc6b768-fd1a-4830-b8db-eaa24e294102" containerID="15167bff4564fc610295e4134629dec685a596ea5e50237651873a2ac1cfa06f" exitCode=0
Nov 03 06:15:31.547867 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:31.547466    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf" event={"ID":"dcc6b768-fd1a-4830-b8db-eaa24e294102","Type":"ContainerDied","Data":"15167bff4564fc610295e4134629dec685a596ea5e50237651873a2ac1cfa06f"}
Nov 03 06:15:31.750074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:31.750035    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2bd2e4c5-6507-44c2-ad7e-066fc230ff69" path="/var/lib/kubelet/pods/2bd2e4c5-6507-44c2-ad7e-066fc230ff69/volumes"
Nov 03 06:15:32.693631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.693607    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:32.765645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.765611    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\" (UniqueName: \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\") pod \"dcc6b768-fd1a-4830-b8db-eaa24e294102\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") "
Nov 03 06:15:32.765851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.765679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\") pod \"dcc6b768-fd1a-4830-b8db-eaa24e294102\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") "
Nov 03 06:15:32.765851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.765832    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\") pod \"dcc6b768-fd1a-4830-b8db-eaa24e294102\" (UID: \"dcc6b768-fd1a-4830-b8db-eaa24e294102\") "
Nov 03 06:15:32.766175 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.766139    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca" pod "dcc6b768-fd1a-4830-b8db-eaa24e294102" (UID: "dcc6b768-fd1a-4830-b8db-eaa24e294102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:15:32.768047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.768015    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52" pod "dcc6b768-fd1a-4830-b8db-eaa24e294102" (UID: "dcc6b768-fd1a-4830-b8db-eaa24e294102"). InnerVolumeSpecName "kube-api-access-5vq52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:15:32.768249 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.768223    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca" pod "dcc6b768-fd1a-4830-b8db-eaa24e294102" (UID: "dcc6b768-fd1a-4830-b8db-eaa24e294102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:15:32.867735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.867646    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vq52\" (UniqueName: \"kubernetes.io/projected/dcc6b768-fd1a-4830-b8db-eaa24e294102-kube-api-access-5vq52\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:32.867735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.867678    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:32.867735 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:32.867691    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcc6b768-fd1a-4830-b8db-eaa24e294102-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:33.563398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:33.563366    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"
Nov 03 06:15:33.563398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:33.563385    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf" event={"ID":"dcc6b768-fd1a-4830-b8db-eaa24e294102","Type":"ContainerDied","Data":"95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25"}
Nov 03 06:15:33.563617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:33.563431    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95eab3392b7e733bd55917b7f24003b0d6d6351764c0949a3ad90af64684cf25"
Nov 03 06:15:42.034770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:42.034732    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"]
Nov 03 06:15:42.046255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:42.046215    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"]
Nov 03 06:15:42.049921 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:42.049889    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6c5a406c-npw7f"]
Nov 03 06:15:42.052948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:42.052918    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-49ea6a2c-whdtz"]
Nov 03 06:15:43.710821 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:15:43.710758    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:15:43.757289 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:43.757250    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0576776c-c783-4fa8-b9ad-9dac7f8c64d2" path="/var/lib/kubelet/pods/0576776c-c783-4fa8-b9ad-9dac7f8c64d2/volumes"
Nov 03 06:15:43.757908 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:43.757885    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17e7983a-9e49-4b04-8be1-a8ab81e780c1" path="/var/lib/kubelet/pods/17e7983a-9e49-4b04-8be1-a8ab81e780c1/volumes"
Nov 03 06:15:44.033485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:44.033377    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"]
Nov 03 06:15:44.045446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:44.045403    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88477545-dmvql"]
Nov 03 06:15:45.775443 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:45.775402    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1fc2a23-1289-4249-9241-52a294fcf009" path="/var/lib/kubelet/pods/c1fc2a23-1289-4249-9241-52a294fcf009/volumes"
Nov 03 06:15:46.133612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:46.133570    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"]
Nov 03 06:15:46.133948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:46.133927    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-934fe5f1-7wl5k"]
Nov 03 06:15:47.747367 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:47.747334    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d224c3-fd32-40c9-84fe-3ec2e2a77e93" path="/var/lib/kubelet/pods/88d224c3-fd32-40c9-84fe-3ec2e2a77e93/volumes"
Nov 03 06:15:49.286631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.286596    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"]
Nov 03 06:15:49.287341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287314    4528 cpu_manager.go:414] "containerMap: removing container" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:49.287341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287339    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:49.287517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287374    4528 cpu_manager.go:414] "containerMap: removing container" podUID="dcc6b768-fd1a-4830-b8db-eaa24e294102" containerName="revoke-keys"
Nov 03 06:15:49.287517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287382    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcc6b768-fd1a-4830-b8db-eaa24e294102" containerName="revoke-keys"
Nov 03 06:15:49.287517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287504    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1eeff51-f66a-4cdc-b677-4175d49b3a5e" containerName="cleanup"
Nov 03 06:15:49.287616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.287521    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcc6b768-fd1a-4830-b8db-eaa24e294102" containerName="revoke-keys"
Nov 03 06:15:49.292975 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.292955    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.308083 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.308058    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"]
Nov 03 06:15:49.314534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.314507    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:15:49.314534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.314526    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:15:49.314684 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.314507    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:15:49.318878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.318859    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:15:49.458647 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.458610    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.458852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.458657    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6894\" (UniqueName: \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.458852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.458827    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.559616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.559579    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.559787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.559645    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.559787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.559686    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6894\" (UniqueName: \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.560360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.560339    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.561946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.561928    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.582855 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.582820    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6894\" (UniqueName: \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\") pod \"maas-api-revoke-sub-0b7e8edb-glg9r\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.609789 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.609765    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:49.910174 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:49.910141    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"]
Nov 03 06:15:49.911060 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:15:49.911033    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ffb9567_2a57_4ec7_acb6_763d2b489911.slice/crio-b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757 WatchSource:0}: Error finding container b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757: Status 404 returned error can't find the container with id b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757
Nov 03 06:15:50.758374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:50.758340    4528 generic.go:356] "Generic (PLEG): container finished" podID="6ffb9567-2a57-4ec7-acb6-763d2b489911" containerID="f6290d82eb991c0ba7fac69c3b7bc7167d3adb5c53c591cbe2f41670ea6e8ad4" exitCode=0
Nov 03 06:15:50.758782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:50.758375    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r" event={"ID":"6ffb9567-2a57-4ec7-acb6-763d2b489911","Type":"ContainerDied","Data":"f6290d82eb991c0ba7fac69c3b7bc7167d3adb5c53c591cbe2f41670ea6e8ad4"}
Nov 03 06:15:50.758782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:50.758432    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r" event={"ID":"6ffb9567-2a57-4ec7-acb6-763d2b489911","Type":"ContainerStarted","Data":"b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757"}
Nov 03 06:15:51.897129 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.897097    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:51.987983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.987947    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\") pod \"6ffb9567-2a57-4ec7-acb6-763d2b489911\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") "
Nov 03 06:15:51.988173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.988038    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\" (UniqueName: \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\") pod \"6ffb9567-2a57-4ec7-acb6-763d2b489911\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") "
Nov 03 06:15:51.988173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.988068    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\") pod \"6ffb9567-2a57-4ec7-acb6-763d2b489911\" (UID: \"6ffb9567-2a57-4ec7-acb6-763d2b489911\") "
Nov 03 06:15:51.988328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.988303    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca" pod "6ffb9567-2a57-4ec7-acb6-763d2b489911" (UID: "6ffb9567-2a57-4ec7-acb6-763d2b489911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:15:51.988464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.988446    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:51.990214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.990185    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894" pod "6ffb9567-2a57-4ec7-acb6-763d2b489911" (UID: "6ffb9567-2a57-4ec7-acb6-763d2b489911"). InnerVolumeSpecName "kube-api-access-b6894". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:15:51.990310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:51.990222    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca" pod "6ffb9567-2a57-4ec7-acb6-763d2b489911" (UID: "6ffb9567-2a57-4ec7-acb6-763d2b489911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:15:52.089302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:52.089260    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6894\" (UniqueName: \"kubernetes.io/projected/6ffb9567-2a57-4ec7-acb6-763d2b489911-kube-api-access-b6894\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:52.089302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:52.089296    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ffb9567-2a57-4ec7-acb6-763d2b489911-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:15:52.775624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:52.775587    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"
Nov 03 06:15:52.775778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:52.775623    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r" event={"ID":"6ffb9567-2a57-4ec7-acb6-763d2b489911","Type":"ContainerDied","Data":"b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757"}
Nov 03 06:15:52.775778 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:52.775678    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b31527bcae826344402c3cca474ba757d83baac982922895e8f74b9ac2f84757"
Nov 03 06:15:57.031017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.030975    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"]
Nov 03 06:15:57.042804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.042765    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"]
Nov 03 06:15:57.046224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.046194    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"]
Nov 03 06:15:57.049110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.049086    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-353d94d9-926nq"]
Nov 03 06:15:57.052026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.051995    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-387aa54e-x7fmp"]
Nov 03 06:15:57.055265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.055242    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-280cfe3e-zctlt"]
Nov 03 06:15:57.748220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.748190    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10d665a3-6269-42b2-b006-2d5e0d9329c3" path="/var/lib/kubelet/pods/10d665a3-6269-42b2-b006-2d5e0d9329c3/volumes"
Nov 03 06:15:57.748634 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.748619    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365f8ed1-62fc-47b1-8818-92c71dd473ef" path="/var/lib/kubelet/pods/365f8ed1-62fc-47b1-8818-92c71dd473ef/volumes"
Nov 03 06:15:57.749048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:15:57.749034    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa4d04ba-d007-4030-985a-50c4cff22772" path="/var/lib/kubelet/pods/aa4d04ba-d007-4030-985a-50c4cff22772/volumes"
Nov 03 06:16:00.045966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.045928    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"]
Nov 03 06:16:00.046401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.046388    4528 cpu_manager.go:414] "containerMap: removing container" podUID="6ffb9567-2a57-4ec7-acb6-763d2b489911" containerName="revoke-keys"
Nov 03 06:16:00.046446 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.046402    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffb9567-2a57-4ec7-acb6-763d2b489911" containerName="revoke-keys"
Nov 03 06:16:00.046496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.046487    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffb9567-2a57-4ec7-acb6-763d2b489911" containerName="revoke-keys"
Nov 03 06:16:00.051201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.051177    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.068273 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.068241    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.068437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.068297    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.068437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.068374    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkm9h\" (UniqueName: \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.068557 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.068539    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"]
Nov 03 06:16:00.075522 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.075495    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:16:00.075731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.075625    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:16:00.075731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.075625    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:16:00.076987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.076957    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:16:00.169284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.169241    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.169481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.169381    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.169481 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.169416    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkm9h\" (UniqueName: \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.170137 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.170101    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.171673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.171649    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.192093 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.192057    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkm9h\" (UniqueName: \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\") pod \"maas-api-revoke-sub-3d4156f1-94jng\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.364777 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.364735    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:00.667140 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.667106    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"]
Nov 03 06:16:00.667971 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:16:00.667937    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c93e1f4_9465_43de_847e_a1513874db6f.slice/crio-017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812 WatchSource:0}: Error finding container 017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812: Status 404 returned error can't find the container with id 017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812
Nov 03 06:16:00.836700 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:00.836659    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng" event={"ID":"0c93e1f4-9465-43de-847e-a1513874db6f","Type":"ContainerStarted","Data":"017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812"}
Nov 03 06:16:01.846631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:01.846541    4528 generic.go:356] "Generic (PLEG): container finished" podID="0c93e1f4-9465-43de-847e-a1513874db6f" containerID="3a4dad2b95888b2140481a375ae4e91dd59050940caab029ce2cd7f885d58767" exitCode=0
Nov 03 06:16:01.847012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:01.846625    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng" event={"ID":"0c93e1f4-9465-43de-847e-a1513874db6f","Type":"ContainerDied","Data":"3a4dad2b95888b2140481a375ae4e91dd59050940caab029ce2cd7f885d58767"}
Nov 03 06:16:03.012609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.012581    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:03.097996 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.097964    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\" (UniqueName: \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\") pod \"0c93e1f4-9465-43de-847e-a1513874db6f\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") "
Nov 03 06:16:03.098152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.098009    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\") pod \"0c93e1f4-9465-43de-847e-a1513874db6f\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") "
Nov 03 06:16:03.098152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.098128    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\") pod \"0c93e1f4-9465-43de-847e-a1513874db6f\" (UID: \"0c93e1f4-9465-43de-847e-a1513874db6f\") "
Nov 03 06:16:03.098416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.098390    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca" pod "0c93e1f4-9465-43de-847e-a1513874db6f" (UID: "0c93e1f4-9465-43de-847e-a1513874db6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:16:03.098596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.098567    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:03.100171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.100135    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h" pod "0c93e1f4-9465-43de-847e-a1513874db6f" (UID: "0c93e1f4-9465-43de-847e-a1513874db6f"). InnerVolumeSpecName "kube-api-access-pkm9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:16:03.100269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.100185    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca" pod "0c93e1f4-9465-43de-847e-a1513874db6f" (UID: "0c93e1f4-9465-43de-847e-a1513874db6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:16:03.199432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.199384    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c93e1f4-9465-43de-847e-a1513874db6f-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:03.199432 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.199427    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkm9h\" (UniqueName: \"kubernetes.io/projected/0c93e1f4-9465-43de-847e-a1513874db6f-kube-api-access-pkm9h\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:03.862385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.862344    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng" event={"ID":"0c93e1f4-9465-43de-847e-a1513874db6f","Type":"ContainerDied","Data":"017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812"}
Nov 03 06:16:03.862569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.862431    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="017932ac26b7928224071b2d58d90f8b4f253ef02d77d7132af51d4f62d89812"
Nov 03 06:16:03.862569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:03.862356    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"
Nov 03 06:16:13.841818 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:16:13.838436    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:16:13.911823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:13.908403    4528 scope.go:122] "RemoveContainer" containerID="8dcb5a08c64d2a9e5a8db3070893920f96c7f2d6cf95efafc817d6bd32319ba9"
Nov 03 06:16:13.966952 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:13.966920    4528 scope.go:122] "RemoveContainer" containerID="0329d448fc587a67837660b15ad54e47f12a55895e38f1d6b663fd3b997932b6"
Nov 03 06:16:14.024447 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.024031    4528 scope.go:122] "RemoveContainer" containerID="9be43a0145d73021f5462bce2d538a29b857d90fdfa3d6399f0970f31e670dfd"
Nov 03 06:16:14.106901 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.106415    4528 scope.go:122] "RemoveContainer" containerID="60f9eb3a53b5542e850686e164f4e14ae52c5361bc63a4c78ca0cd2a6b44192e"
Nov 03 06:16:14.257316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.257165    4528 scope.go:122] "RemoveContainer" containerID="7a8089ec9d28335b73c4b5d1a32ad28b3014c712923b7c9f2fb87abcd78fd414"
Nov 03 06:16:14.313842 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.307046    4528 scope.go:122] "RemoveContainer" containerID="06b04436640f726c514106dff4678cabeec2770db73af11edbd129e7d87dc067"
Nov 03 06:16:14.383418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.382942    4528 scope.go:122] "RemoveContainer" containerID="47254c3d8f879634fe31882132347e50ad608b4700f2fa47f3bd897ebbc6c691"
Nov 03 06:16:14.473293 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.473114    4528 scope.go:122] "RemoveContainer" containerID="687289b84ff7b7520365add6df41535f69f0fe124e86582d5c8d715904ba122e"
Nov 03 06:16:14.518715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.518582    4528 scope.go:122] "RemoveContainer" containerID="4bc5f5d8c479c0992a7ef6919c578ce5ba1dab7982550564456bf5f30bbf7cd9"
Nov 03 06:16:14.565110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.564953    4528 scope.go:122] "RemoveContainer" containerID="d7f80d10eb411b3bf157ca703cfe73786d323922c5df6872ae69ef705d93a532"
Nov 03 06:16:14.581516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.581125    4528 scope.go:122] "RemoveContainer" containerID="afc2f158f54d08e22d179f46b25e056bad76109efcb72f98e794849296521b01"
Nov 03 06:16:14.641402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.640675    4528 scope.go:122] "RemoveContainer" containerID="8c793e6e9c48cd49724e71024f9931252079ea8a2ce4c2891f0f760cd32c3ccc"
Nov 03 06:16:14.676645 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.676513    4528 scope.go:122] "RemoveContainer" containerID="0e37e03ebd46fbeb779f90187a78955497fd07ab682bc852bd023e23c0b0b13d"
Nov 03 06:16:14.697152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:14.696768    4528 scope.go:122] "RemoveContainer" containerID="2aae61c0dea394082fbc40faa7d0e8e25be632a70831dc2c8fc34622e727b9a1"
Nov 03 06:16:16.074970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:16.074870    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"]
Nov 03 06:16:16.145951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:16.145911    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b6e5b33b-kfznl"]
Nov 03 06:16:16.161065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:16.161025    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"]
Nov 03 06:16:16.183831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:16.165403    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6428dff-xsspb"]
Nov 03 06:16:17.785818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:17.785309    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23cb76eb-d1c9-4b47-afca-ad805468b5ca" path="/var/lib/kubelet/pods/23cb76eb-d1c9-4b47-afca-ad805468b5ca/volumes"
Nov 03 06:16:17.786317 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:17.785895    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="244f3d91-8ec4-4b62-bc20-585dd79e9809" path="/var/lib/kubelet/pods/244f3d91-8ec4-4b62-bc20-585dd79e9809/volumes"
Nov 03 06:16:23.643024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.642985    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"]
Nov 03 06:16:23.654221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.654190    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-94jng"]
Nov 03 06:16:23.696650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.696608    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"]
Nov 03 06:16:23.697284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.697163    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0c93e1f4-9465-43de-847e-a1513874db6f" containerName="revoke-keys"
Nov 03 06:16:23.697284 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.697185    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c93e1f4-9465-43de-847e-a1513874db6f" containerName="revoke-keys"
Nov 03 06:16:23.697472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.697316    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c93e1f4-9465-43de-847e-a1513874db6f" containerName="revoke-keys"
Nov 03 06:16:23.704132 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.704108    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.718584 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.718560    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:16:23.718872 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.718836    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:16:23.719002 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.718976    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:16:23.719150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.719125    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:16:23.719520 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.719500    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"]
Nov 03 06:16:23.748971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.748940    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c93e1f4-9465-43de-847e-a1513874db6f" path="/var/lib/kubelet/pods/0c93e1f4-9465-43de-847e-a1513874db6f/volumes"
Nov 03 06:16:23.841216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.841172    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.841371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.841248    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtqdf\" (UniqueName: \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.841371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.841279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.941868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.941755    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtqdf\" (UniqueName: \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.941868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.941836    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.942120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.941905    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.942542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.942516    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.944365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.944342    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:23.964533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:23.964498    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtqdf\" (UniqueName: \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\") pod \"maas-api-revoke-sub-3d4156f1-4dbvh\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:24.020070 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.020034    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:24.030670 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.030636    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"]
Nov 03 06:16:24.034699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.034666    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"]
Nov 03 06:16:24.038490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.038464    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227fa5ae-p57gq"]
Nov 03 06:16:24.041404 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.041379    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cdc8d99f-n47fl"]
Nov 03 06:16:24.325449 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:24.325416    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"]
Nov 03 06:16:24.326547 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:16:24.326516    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b50d3d_4732_445b_a00e_c8772f22fc3d.slice/crio-11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e WatchSource:0}: Error finding container 11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e: Status 404 returned error can't find the container with id 11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e
Nov 03 06:16:25.181061 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:25.181028    4528 generic.go:356] "Generic (PLEG): container finished" podID="f5b50d3d-4732-445b-a00e-c8772f22fc3d" containerID="257577aabbf9a42dbca8e51664d7112cda248d939821feea81ab0337b901ae4f" exitCode=0
Nov 03 06:16:25.181612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:25.181074    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh" event={"ID":"f5b50d3d-4732-445b-a00e-c8772f22fc3d","Type":"ContainerDied","Data":"257577aabbf9a42dbca8e51664d7112cda248d939821feea81ab0337b901ae4f"}
Nov 03 06:16:25.181612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:25.181176    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh" event={"ID":"f5b50d3d-4732-445b-a00e-c8772f22fc3d","Type":"ContainerStarted","Data":"11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e"}
Nov 03 06:16:25.749303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:25.749258    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb6d8e8-20ff-44f0-a158-592cc43461ff" path="/var/lib/kubelet/pods/6cb6d8e8-20ff-44f0-a158-592cc43461ff/volumes"
Nov 03 06:16:25.749807 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:25.749770    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb3b4a50-0498-4d01-894c-591c037e18f2" path="/var/lib/kubelet/pods/fb3b4a50-0498-4d01-894c-591c037e18f2/volumes"
Nov 03 06:16:26.337965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.337934    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:26.471296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.471214    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\") pod \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") "
Nov 03 06:16:26.471296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.471290    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\" (UniqueName: \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\") pod \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") "
Nov 03 06:16:26.471487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.471315    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\") pod \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\" (UID: \"f5b50d3d-4732-445b-a00e-c8772f22fc3d\") "
Nov 03 06:16:26.471757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.471722    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca" pod "f5b50d3d-4732-445b-a00e-c8772f22fc3d" (UID: "f5b50d3d-4732-445b-a00e-c8772f22fc3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:16:26.473471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.473445    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca" pod "f5b50d3d-4732-445b-a00e-c8772f22fc3d" (UID: "f5b50d3d-4732-445b-a00e-c8772f22fc3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:16:26.473471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.473457    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf" pod "f5b50d3d-4732-445b-a00e-c8772f22fc3d" (UID: "f5b50d3d-4732-445b-a00e-c8772f22fc3d"). InnerVolumeSpecName "kube-api-access-vtqdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:16:26.572612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.572571    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:26.572612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.572605    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5b50d3d-4732-445b-a00e-c8772f22fc3d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:26.572612 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:26.572614    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtqdf\" (UniqueName: \"kubernetes.io/projected/f5b50d3d-4732-445b-a00e-c8772f22fc3d-kube-api-access-vtqdf\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:16:27.196643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:27.196610    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"
Nov 03 06:16:27.196851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:27.196641    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh" event={"ID":"f5b50d3d-4732-445b-a00e-c8772f22fc3d","Type":"ContainerDied","Data":"11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e"}
Nov 03 06:16:27.196851 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:27.196692    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="11435ce35755c06581c3374d2e4babb8cc9226f65a482ae34ed8ddd92538683e"
Nov 03 06:16:43.692439 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:16:43.692401    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:16:59.792861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.792749    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"]
Nov 03 06:16:59.793471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.793228    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f5b50d3d-4732-445b-a00e-c8772f22fc3d" containerName="revoke-keys"
Nov 03 06:16:59.793471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.793241    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b50d3d-4732-445b-a00e-c8772f22fc3d" containerName="revoke-keys"
Nov 03 06:16:59.793471 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.793314    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b50d3d-4732-445b-a00e-c8772f22fc3d" containerName="revoke-keys"
Nov 03 06:16:59.797158 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.797137    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.815682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.815639    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"]
Nov 03 06:16:59.822668 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.822635    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:16:59.824266 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.824239    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:16:59.824380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.824348    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:16:59.824473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.824450    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:16:59.856809 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.856766    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzj7z\" (UniqueName: \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.856970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.856846    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.856970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.856948    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.957967 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.957926    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.958145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.958062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzj7z\" (UniqueName: \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.958145 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.958085    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.958706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.958675    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.960303 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.960282    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:16:59.981042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:16:59.981019    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzj7z\" (UniqueName: \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\") pod \"maas-api-revoke-sub-e2fc9c5b-dgxg2\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:17:00.115419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:00.115380    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:17:00.414511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:00.414485    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"]
Nov 03 06:17:00.415954 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:17:00.415911    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda394fca_7901_42cc_82bc_e360e6c2efa3.slice/crio-b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e WatchSource:0}: Error finding container b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e: Status 404 returned error can't find the container with id b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e
Nov 03 06:17:00.464979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:00.464941    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2" event={"ID":"da394fca-7901-42cc-82bc-e360e6c2efa3","Type":"ContainerStarted","Data":"b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e"}
Nov 03 06:17:01.473965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:01.473933    4528 generic.go:356] "Generic (PLEG): container finished" podID="da394fca-7901-42cc-82bc-e360e6c2efa3" containerID="0e9b29ea3722898479d0c78b57f00da48703e9c2789dd15249c09d56ad4ba237" exitCode=0
Nov 03 06:17:01.474371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:01.473988    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2" event={"ID":"da394fca-7901-42cc-82bc-e360e6c2efa3","Type":"ContainerDied","Data":"0e9b29ea3722898479d0c78b57f00da48703e9c2789dd15249c09d56ad4ba237"}
Nov 03 06:17:02.623360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.623327    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:17:02.685935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.685888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\") pod \"da394fca-7901-42cc-82bc-e360e6c2efa3\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") "
Nov 03 06:17:02.686130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.686078    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\") pod \"da394fca-7901-42cc-82bc-e360e6c2efa3\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") "
Nov 03 06:17:02.686130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.686102    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\" (UniqueName: \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\") pod \"da394fca-7901-42cc-82bc-e360e6c2efa3\" (UID: \"da394fca-7901-42cc-82bc-e360e6c2efa3\") "
Nov 03 06:17:02.686269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.686244    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca" pod "da394fca-7901-42cc-82bc-e360e6c2efa3" (UID: "da394fca-7901-42cc-82bc-e360e6c2efa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:17:02.686377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.686362    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:02.688431 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.688396    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z" pod "da394fca-7901-42cc-82bc-e360e6c2efa3" (UID: "da394fca-7901-42cc-82bc-e360e6c2efa3"). InnerVolumeSpecName "kube-api-access-rzj7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:17:02.688540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.688479    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca" pod "da394fca-7901-42cc-82bc-e360e6c2efa3" (UID: "da394fca-7901-42cc-82bc-e360e6c2efa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:17:02.787244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.787158    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da394fca-7901-42cc-82bc-e360e6c2efa3-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:02.787244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:02.787190    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzj7z\" (UniqueName: \"kubernetes.io/projected/da394fca-7901-42cc-82bc-e360e6c2efa3-kube-api-access-rzj7z\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:03.489238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:03.489209    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"
Nov 03 06:17:03.489423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:03.489245    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2" event={"ID":"da394fca-7901-42cc-82bc-e360e6c2efa3","Type":"ContainerDied","Data":"b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e"}
Nov 03 06:17:03.489423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:03.489290    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b40d3ce495a6001ec5bd5733ccbca1654f15010ed4b61ad52818482eba1cec9e"
Nov 03 06:17:10.140189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.140129    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"]
Nov 03 06:17:10.140820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.140684    4528 cpu_manager.go:414] "containerMap: removing container" podUID="da394fca-7901-42cc-82bc-e360e6c2efa3" containerName="revoke-keys"
Nov 03 06:17:10.140820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.140699    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da394fca-7901-42cc-82bc-e360e6c2efa3" containerName="revoke-keys"
Nov 03 06:17:10.140820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.140788    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="da394fca-7901-42cc-82bc-e360e6c2efa3" containerName="revoke-keys"
Nov 03 06:17:10.144108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.144085    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.164590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.164553    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"]
Nov 03 06:17:10.170173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.170146    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:17:10.170410 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.170170    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:17:10.170545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.170202    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:17:10.170644 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.170234    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:17:10.266238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.266201    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.266413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.266321    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtrdk\" (UniqueName: \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.266413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.266344    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.367536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.367491    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtrdk\" (UniqueName: \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.367536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.367539    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.367766 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.367606    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.368281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.368246    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.370044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.370023    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.391077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.390993    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtrdk\" (UniqueName: \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\") pod \"maas-api-revoke-sub-5a29efb3-s29vl\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.461268 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.461228    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:10.765498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:10.765467    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"]
Nov 03 06:17:10.767060 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:17:10.767029    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7999fed9_5e74_4518_9c32_c23758d989a5.slice/crio-24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807 WatchSource:0}: Error finding container 24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807: Status 404 returned error can't find the container with id 24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807
Nov 03 06:17:11.555428 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:11.555390    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl" event={"ID":"7999fed9-5e74-4518-9c32-c23758d989a5","Type":"ContainerStarted","Data":"f51b5d3dc0fe1f22b432828dfb3fef691bfa43fb81fd67a3d858d042bb8bac03"}
Nov 03 06:17:11.555991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:11.555446    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl" event={"ID":"7999fed9-5e74-4518-9c32-c23758d989a5","Type":"ContainerStarted","Data":"24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807"}
Nov 03 06:17:11.603059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:11.602625    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl" podStartSLOduration=1.60260717 podStartE2EDuration="1.60260717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:17:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:17:11.601773617 +0000 UTC m=+1320.013170699" watchObservedRunningTime="2026-11-03 06:17:11.60260717 +0000 UTC m=+1320.014004249"
Nov 03 06:17:12.032934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:12.032816    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"]
Nov 03 06:17:12.038440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:12.038396    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-swrcl"]
Nov 03 06:17:12.596218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:12.595765    4528 generic.go:356] "Generic (PLEG): container finished" podID="7999fed9-5e74-4518-9c32-c23758d989a5" containerID="f51b5d3dc0fe1f22b432828dfb3fef691bfa43fb81fd67a3d858d042bb8bac03" exitCode=0
Nov 03 06:17:12.596729 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:12.595976    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl" event={"ID":"7999fed9-5e74-4518-9c32-c23758d989a5","Type":"ContainerDied","Data":"f51b5d3dc0fe1f22b432828dfb3fef691bfa43fb81fd67a3d858d042bb8bac03"}
Nov 03 06:17:13.679458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:13.677906    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:17:13.679458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:13.677978    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:17:13.852819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:13.852120    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0e8af1-c978-43a0-9034-ebe162d75c87" path="/var/lib/kubelet/pods/1d0e8af1-c978-43a0-9034-ebe162d75c87/volumes"
Nov 03 06:17:14.166724 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:17:14.166631    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:17:14.394818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.394053    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:14.493816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.493212    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\") pod \"7999fed9-5e74-4518-9c32-c23758d989a5\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") "
Nov 03 06:17:14.493816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.493352    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\" (UniqueName: \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\") pod \"7999fed9-5e74-4518-9c32-c23758d989a5\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") "
Nov 03 06:17:14.493816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.493427    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\") pod \"7999fed9-5e74-4518-9c32-c23758d989a5\" (UID: \"7999fed9-5e74-4518-9c32-c23758d989a5\") "
Nov 03 06:17:14.517690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.517651    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca" pod "7999fed9-5e74-4518-9c32-c23758d989a5" (UID: "7999fed9-5e74-4518-9c32-c23758d989a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:17:14.530874 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.529986    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca" pod "7999fed9-5e74-4518-9c32-c23758d989a5" (UID: "7999fed9-5e74-4518-9c32-c23758d989a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:17:14.577824 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.571061    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk" pod "7999fed9-5e74-4518-9c32-c23758d989a5" (UID: "7999fed9-5e74-4518-9c32-c23758d989a5"). InnerVolumeSpecName "kube-api-access-wtrdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:17:14.597422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.597307    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:14.597422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.597353    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtrdk\" (UniqueName: \"kubernetes.io/projected/7999fed9-5e74-4518-9c32-c23758d989a5-kube-api-access-wtrdk\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:14.597422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.597391    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7999fed9-5e74-4518-9c32-c23758d989a5-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:14.633579 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.633547    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"
Nov 03 06:17:14.633750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.633539    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl" event={"ID":"7999fed9-5e74-4518-9c32-c23758d989a5","Type":"ContainerDied","Data":"24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807"}
Nov 03 06:17:14.633750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.633695    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24452eec8849cf87f63530d1b003990845e106800073be500c48bd45e2678807"
Nov 03 06:17:14.855917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.855880    4528 scope.go:122] "RemoveContainer" containerID="6df1fb033685356930368e4e7379fea510270d88ff2a69eb3fdacc53b28a9c67"
Nov 03 06:17:14.869485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.869459    4528 scope.go:122] "RemoveContainer" containerID="a88f6cf4552ec7fa4cc1d2cb466302ff3690a0abac0c5e29fd76a03743606893"
Nov 03 06:17:14.879725 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.879700    4528 scope.go:122] "RemoveContainer" containerID="8aae7550d30764f3cbf374776c76ab0c077bb270c946e01914efd511f95932bd"
Nov 03 06:17:14.889357 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.889329    4528 scope.go:122] "RemoveContainer" containerID="9357775e4f2ff911043a3f8993095da269687716fab1d7912ef1aa792b637fd7"
Nov 03 06:17:14.902626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.902603    4528 scope.go:122] "RemoveContainer" containerID="10f9363668ce4bb735165aabf9b665dd01d22bb676d1efada0e53d255fbaf00a"
Nov 03 06:17:14.912136 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.912114    4528 scope.go:122] "RemoveContainer" containerID="f5a55b4dbd963a44cee2810a3e9f239fb1bfa6b8ccac74e7eab360736ddfc15b"
Nov 03 06:17:14.923236 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.923211    4528 scope.go:122] "RemoveContainer" containerID="4a726ac5e467897dec5cfa0223961ad16fa6151e1238ec2b0d7c88f873672860"
Nov 03 06:17:14.933238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.933215    4528 scope.go:122] "RemoveContainer" containerID="c82cc01859153af0a29d1e872f97296072983530e788897ac7488e385bca2dfa"
Nov 03 06:17:14.944600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.944574    4528 scope.go:122] "RemoveContainer" containerID="07973c529ce482929bfc3ef559b47e1bf7c6b187c4e1ebb1f6ca67544a53ff9c"
Nov 03 06:17:14.955943 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.955907    4528 scope.go:122] "RemoveContainer" containerID="f4a486325fcc72cb4fc552535d1eb8ccdb2c91fda1a2369abc9cec7bd616bd7c"
Nov 03 06:17:14.965633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.965613    4528 scope.go:122] "RemoveContainer" containerID="628f03adb0f2a660cbceecaf1015fdc096529d1d9f3fb9e1fdf9c21d8314f7be"
Nov 03 06:17:14.975078 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.975055    4528 scope.go:122] "RemoveContainer" containerID="3bdf32eb8784a8bc1d7213d94b3bc8f08998684fd0c41c8aa3a94c6fd9249abb"
Nov 03 06:17:14.984074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.984055    4528 scope.go:122] "RemoveContainer" containerID="f4c9743cc2ccfdfc8d6f83032baa6899444a76e8da8061eac751b16fac472496"
Nov 03 06:17:14.993525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:14.993502    4528 scope.go:122] "RemoveContainer" containerID="b990aa1f89200b254d6e0a1da15c9b9e23ed6365b953b67e29a1fb57d1efbf4e"
Nov 03 06:17:15.003350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:15.003328    4528 scope.go:122] "RemoveContainer" containerID="74758de8f7be49e6b92d7b5fa78fe680661177d8744d24d5fded4c7181038790"
Nov 03 06:17:15.014999 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:15.014973    4528 scope.go:122] "RemoveContainer" containerID="652bd1b5af0fdc5aed6c0d1e36c3486a81922eb9794cbd9e74c36254340035c0"
Nov 03 06:17:28.356658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.356616    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"]
Nov 03 06:17:28.368377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.368345    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4dbvh"]
Nov 03 06:17:28.407651 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.407618    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"]
Nov 03 06:17:28.408240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.408218    4528 cpu_manager.go:414] "containerMap: removing container" podUID="7999fed9-5e74-4518-9c32-c23758d989a5" containerName="revoke-keys"
Nov 03 06:17:28.408240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.408240    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7999fed9-5e74-4518-9c32-c23758d989a5" containerName="revoke-keys"
Nov 03 06:17:28.408388 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.408321    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="7999fed9-5e74-4518-9c32-c23758d989a5" containerName="revoke-keys"
Nov 03 06:17:28.415500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.415474    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.430510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.430475    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:17:28.430705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.430508    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:17:28.430705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.430576    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"]
Nov 03 06:17:28.430705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.430583    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:17:28.430705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.430588    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:17:28.450195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.450157    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.450376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.450250    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.450376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.450314    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t68js\" (UniqueName: \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.551176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.551132    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.551176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.551178    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.551462 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.551207    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t68js\" (UniqueName: \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.551965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.551938    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.553626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.553601    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.574331 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.574301    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t68js\" (UniqueName: \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\") pod \"maas-api-revoke-sub-3d4156f1-d28jd\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:28.730706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:28.730620    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:29.035903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:29.035866    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"]
Nov 03 06:17:29.037008 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:17:29.036974    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dda55bf_6239_4a99_b133_959eb89a0ffb.slice/crio-64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5 WatchSource:0}: Error finding container 64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5: Status 404 returned error can't find the container with id 64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5
Nov 03 06:17:29.738460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:29.738427    4528 generic.go:356] "Generic (PLEG): container finished" podID="2dda55bf-6239-4a99-b133-959eb89a0ffb" containerID="1974c4b52a0fd771cb9e255dd4f1820bfec9d08dd95603dbfe532b4bca0c899a" exitCode=0
Nov 03 06:17:29.738932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:29.738512    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd" event={"ID":"2dda55bf-6239-4a99-b133-959eb89a0ffb","Type":"ContainerDied","Data":"1974c4b52a0fd771cb9e255dd4f1820bfec9d08dd95603dbfe532b4bca0c899a"}
Nov 03 06:17:29.738932 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:29.738571    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd" event={"ID":"2dda55bf-6239-4a99-b133-959eb89a0ffb","Type":"ContainerStarted","Data":"64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5"}
Nov 03 06:17:29.747804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:29.747759    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b50d3d-4732-445b-a00e-c8772f22fc3d" path="/var/lib/kubelet/pods/f5b50d3d-4732-445b-a00e-c8772f22fc3d/volumes"
Nov 03 06:17:30.885130 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.885101    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:30.974302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.974268    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\") pod \"2dda55bf-6239-4a99-b133-959eb89a0ffb\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") "
Nov 03 06:17:30.974533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.974358    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\") pod \"2dda55bf-6239-4a99-b133-959eb89a0ffb\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") "
Nov 03 06:17:30.974533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.974393    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\" (UniqueName: \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\") pod \"2dda55bf-6239-4a99-b133-959eb89a0ffb\" (UID: \"2dda55bf-6239-4a99-b133-959eb89a0ffb\") "
Nov 03 06:17:30.974733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.974701    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca" pod "2dda55bf-6239-4a99-b133-959eb89a0ffb" (UID: "2dda55bf-6239-4a99-b133-959eb89a0ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:17:30.976510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.976480    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca" pod "2dda55bf-6239-4a99-b133-959eb89a0ffb" (UID: "2dda55bf-6239-4a99-b133-959eb89a0ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:17:30.976639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:30.976595    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js" pod "2dda55bf-6239-4a99-b133-959eb89a0ffb" (UID: "2dda55bf-6239-4a99-b133-959eb89a0ffb"). InnerVolumeSpecName "kube-api-access-t68js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:17:31.075884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.075841    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:31.075884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.075881    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dda55bf-6239-4a99-b133-959eb89a0ffb-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:31.075884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.075892    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t68js\" (UniqueName: \"kubernetes.io/projected/2dda55bf-6239-4a99-b133-959eb89a0ffb-kube-api-access-t68js\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:17:31.752381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.752347    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd" event={"ID":"2dda55bf-6239-4a99-b133-959eb89a0ffb","Type":"ContainerDied","Data":"64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5"}
Nov 03 06:17:31.752595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.752393    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64c05e4e23f0b56877edd98a5171299a0ce7ba21c59707caa1090ac2bc636cb5"
Nov 03 06:17:31.752595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:31.752363    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"
Nov 03 06:17:43.693295 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:17:43.693252    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:17:55.332474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.332431    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:17:55.332983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.332919    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2dda55bf-6239-4a99-b133-959eb89a0ffb" containerName="revoke-keys"
Nov 03 06:17:55.332983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.332933    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dda55bf-6239-4a99-b133-959eb89a0ffb" containerName="revoke-keys"
Nov 03 06:17:55.333064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.333031    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dda55bf-6239-4a99-b133-959eb89a0ffb" containerName="revoke-keys"
Nov 03 06:17:55.336285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.336264    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.359654 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.359605    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:17:55.365343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.365311    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-fcbaef\""
Nov 03 06:17:55.388501 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.388459    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-55h6n\" (UniqueName: \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.388701 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.388581    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.400019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.399981    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:17:55.403850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.403827    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.422541 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.422507    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:17:55.429946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.429918    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-01fa7818-dockercfg-lmgfw\""
Nov 03 06:17:55.491877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.491837    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-55h6n\" (UniqueName: \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.493244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.493025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.495344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.494431    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.515465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.515435    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-55h6n\" (UniqueName: \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\") pod \"payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.596364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.596263    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kwlw\" (UniqueName: \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.596535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.596436    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.649329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.649294    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:55.697616 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.697577    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kwlw\" (UniqueName: \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.697838 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.697697    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.698402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.698380    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.723445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.723411    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kwlw\" (UniqueName: \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\") pod \"payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:55.956282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:55.956252    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:17:55.956863 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:17:55.956833    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb391bc86_1250_4b19_b174_12a02dabd774.slice/crio-271cbfd4f8c59550a7b259bac2c92ee832094a11a8d338cb29594481522fca5f WatchSource:0}: Error finding container 271cbfd4f8c59550a7b259bac2c92ee832094a11a8d338cb29594481522fca5f: Status 404 returned error can't find the container with id 271cbfd4f8c59550a7b259bac2c92ee832094a11a8d338cb29594481522fca5f
Nov 03 06:17:56.017802 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.017765    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:56.317242 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.317208    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:17:56.318412 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:17:56.318378    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod919e6e4e_fb34_4c6a_89c8_201fabda5bf1.slice/crio-d8e1d0ef0a10c62e8d3554be8e2da8b55c79e24c1b3a1d8c41ef23c4abf0f422 WatchSource:0}: Error finding container d8e1d0ef0a10c62e8d3554be8e2da8b55c79e24c1b3a1d8c41ef23c4abf0f422: Status 404 returned error can't find the container with id d8e1d0ef0a10c62e8d3554be8e2da8b55c79e24c1b3a1d8c41ef23c4abf0f422
Nov 03 06:17:56.954494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.954454    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" event={"ID":"b391bc86-1250-4b19-b174-12a02dabd774","Type":"ContainerStarted","Data":"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"}
Nov 03 06:17:56.955028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.954527    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" event={"ID":"b391bc86-1250-4b19-b174-12a02dabd774","Type":"ContainerStarted","Data":"271cbfd4f8c59550a7b259bac2c92ee832094a11a8d338cb29594481522fca5f"}
Nov 03 06:17:56.955028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.954716    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:17:56.956391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.956366    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" event={"ID":"919e6e4e-fb34-4c6a-89c8-201fabda5bf1","Type":"ContainerStarted","Data":"2442531b4d8d2267325d4b0a501b1ffba83748fe70de537177d71ce305c7b4e4"}
Nov 03 06:17:56.956523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.956401    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" event={"ID":"919e6e4e-fb34-4c6a-89c8-201fabda5bf1","Type":"ContainerStarted","Data":"d8e1d0ef0a10c62e8d3554be8e2da8b55c79e24c1b3a1d8c41ef23c4abf0f422"}
Nov 03 06:17:56.956523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:56.956423    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:17:57.002103 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:57.002040    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" podStartSLOduration=2.002025803 podStartE2EDuration="2.002025803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:17:57.0009659 +0000 UTC m=+1365.412362981" watchObservedRunningTime="2026-11-03 06:17:57.002025803 +0000 UTC m=+1365.413422881"
Nov 03 06:17:57.048014 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:17:57.047958    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" podStartSLOduration=2.047942032 podStartE2EDuration="2.047942032s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:17:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:17:57.046583924 +0000 UTC m=+1365.457981004" watchObservedRunningTime="2026-11-03 06:17:57.047942032 +0000 UTC m=+1365.459339110"
Nov 03 06:18:07.968245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:07.968209    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:18:07.969131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:07.968281    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:18:10.034961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.034923    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"]
Nov 03 06:18:10.047484 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.047449    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-tccwx"]
Nov 03 06:18:10.621330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.621293    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:18:10.625486 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.625453    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.643614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.643576    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:18:10.650886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.650855    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fcbaef\"/\"kube-root-ca.crt\""
Nov 03 06:18:10.651075 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.650880    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fcbaef\"/\"simulator-single-main-fcbaef-kserve-self-signed-certs\""
Nov 03 06:18:10.651075 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.650965    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-fcbaef\"/\"openshift-service-ca.crt\""
Nov 03 06:18:10.653878 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.653856    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-fcbaef\"/\"default-dockercfg-fjl6q\""
Nov 03 06:18:10.787570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787448    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.787570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787504    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.787570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787523    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.787924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787720    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c66dp\" (UniqueName: \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.787924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787770    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.787924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.787895    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.889624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889523    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c66dp\" (UniqueName: \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.889624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889569    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.889624 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889602    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889656    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889704    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.889740    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.890178    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890305 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.890276    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.890394 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.890376    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.892098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.892066    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.892351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.892334    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.912592 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.912560    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c66dp\" (UniqueName: \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\") pod \"simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") " pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:10.939621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:10.939582    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:11.243468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:11.243434    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:18:11.244482 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:18:11.244448    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba88cff6_ffb9_4e7c_9a0b_1b6627dd3281.slice/crio-17c7aa87e9c956ff98c23e30ddbeef32369e90bef3cb338b6ca34465fdd797ed WatchSource:0}: Error finding container 17c7aa87e9c956ff98c23e30ddbeef32369e90bef3cb338b6ca34465fdd797ed: Status 404 returned error can't find the container with id 17c7aa87e9c956ff98c23e30ddbeef32369e90bef3cb338b6ca34465fdd797ed
Nov 03 06:18:11.756387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:11.756350    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1ece506-39ed-45db-b463-4869fa461834" path="/var/lib/kubelet/pods/d1ece506-39ed-45db-b463-4869fa461834/volumes"
Nov 03 06:18:12.065159 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:12.065118    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" event={"ID":"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281","Type":"ContainerStarted","Data":"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"}
Nov 03 06:18:12.065373 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:12.065169    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" event={"ID":"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281","Type":"ContainerStarted","Data":"17c7aa87e9c956ff98c23e30ddbeef32369e90bef3cb338b6ca34465fdd797ed"}
Nov 03 06:18:12.111766 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:12.111707    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" podStartSLOduration=2.111693486 podStartE2EDuration="2.111693486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:18:12.110030631 +0000 UTC m=+1380.521427711" watchObservedRunningTime="2026-11-03 06:18:12.111693486 +0000 UTC m=+1380.523090563"
Nov 03 06:18:12.939915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:12.939870    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:13.693291 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:18:13.693251    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:18:15.126707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:15.126673    4528 scope.go:122] "RemoveContainer" containerID="4a048c79dea282868940c31ed9d5983da55c2b8bf112dd199ffdb05751f27167"
Nov 03 06:18:15.137228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:15.137201    4528 scope.go:122] "RemoveContainer" containerID="69c3294d791a7e98e1b51aebdf539257321d45e03296a24f785cc6a1c2cbe3a0"
Nov 03 06:18:15.146947 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:15.146918    4528 scope.go:122] "RemoveContainer" containerID="dc5b92a7b28b8b6bd7b976df557a0867331d4a7682e4968d9c7d73269d1659ab"
Nov 03 06:18:20.940182 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:20.940134    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:20.952955 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:20.952924    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:21.137847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:21.137268    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:18:35.032632 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:35.032413    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"]
Nov 03 06:18:35.041765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:35.041730    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-d6lpl"]
Nov 03 06:18:35.748109 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:35.748074    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b759b3-e1c9-4313-8d53-0d263faa57ef" path="/var/lib/kubelet/pods/07b759b3-e1c9-4313-8d53-0d263faa57ef/volumes"
Nov 03 06:18:42.570523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.570485    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"]
Nov 03 06:18:42.574494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.574474    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.592922 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.592895    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"]
Nov 03 06:18:42.600392 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.600365    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:18:42.600538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.600407    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:18:42.600538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.600492    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:18:42.600538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.600494    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:18:42.736272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.736228    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.736460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.736375    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8mfw\" (UniqueName: \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.736528 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.736512    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.837387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.837299    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.837387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.837374    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8mfw\" (UniqueName: \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.837590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.837430    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.838184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.838156    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.839637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.839619    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.860310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.860283    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8mfw\" (UniqueName: \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\") pod \"maas-api-revoke-sub-2dfc7248-xljgn\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:42.887771 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:42.887746    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:43.185415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:43.185388    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"]
Nov 03 06:18:43.186466 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:18:43.186436    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907d069a_e8b4_413f_bc7a_bf4b937730c3.slice/crio-1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f WatchSource:0}: Error finding container 1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f: Status 404 returned error can't find the container with id 1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f
Nov 03 06:18:43.281276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:43.281241    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn" event={"ID":"907d069a-e8b4-413f-bc7a-bf4b937730c3","Type":"ContainerStarted","Data":"1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f"}
Nov 03 06:18:43.693697 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:18:43.693662    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:18:44.290376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:44.290344    4528 generic.go:356] "Generic (PLEG): container finished" podID="907d069a-e8b4-413f-bc7a-bf4b937730c3" containerID="75c15061f7ca7123add8eb7c5add0b9a45a315db702d2ea3ec9857f7c1ef7a49" exitCode=0
Nov 03 06:18:44.290542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:44.290443    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn" event={"ID":"907d069a-e8b4-413f-bc7a-bf4b937730c3","Type":"ContainerDied","Data":"75c15061f7ca7123add8eb7c5add0b9a45a315db702d2ea3ec9857f7c1ef7a49"}
Nov 03 06:18:45.438077 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.438052    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:45.565634 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.565608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\") pod \"907d069a-e8b4-413f-bc7a-bf4b937730c3\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") "
Nov 03 06:18:45.565820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.565659    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\" (UniqueName: \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\") pod \"907d069a-e8b4-413f-bc7a-bf4b937730c3\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") "
Nov 03 06:18:45.565820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.565740    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\") pod \"907d069a-e8b4-413f-bc7a-bf4b937730c3\" (UID: \"907d069a-e8b4-413f-bc7a-bf4b937730c3\") "
Nov 03 06:18:45.566101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.566076    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca" pod "907d069a-e8b4-413f-bc7a-bf4b937730c3" (UID: "907d069a-e8b4-413f-bc7a-bf4b937730c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:18:45.566224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.566207    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:45.567774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.567747    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca" pod "907d069a-e8b4-413f-bc7a-bf4b937730c3" (UID: "907d069a-e8b4-413f-bc7a-bf4b937730c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:18:45.567961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.567943    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw" pod "907d069a-e8b4-413f-bc7a-bf4b937730c3" (UID: "907d069a-e8b4-413f-bc7a-bf4b937730c3"). InnerVolumeSpecName "kube-api-access-b8mfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:18:45.667086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.667032    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/907d069a-e8b4-413f-bc7a-bf4b937730c3-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:45.667086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:45.667079    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8mfw\" (UniqueName: \"kubernetes.io/projected/907d069a-e8b4-413f-bc7a-bf4b937730c3-kube-api-access-b8mfw\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:46.077534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:46.077500    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"]
Nov 03 06:18:46.088697 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:46.088668    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-mwdpd"]
Nov 03 06:18:46.304374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:46.304343    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"
Nov 03 06:18:46.304374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:46.304348    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn" event={"ID":"907d069a-e8b4-413f-bc7a-bf4b937730c3","Type":"ContainerDied","Data":"1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f"}
Nov 03 06:18:46.304590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:46.304400    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc90e281a1f483f95db0438a12fbacf1e3885bd45c65260be3e619b012e805f"
Nov 03 06:18:47.748089 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:47.748054    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c342fcbd-bae7-4776-9e79-41e49b647daa" path="/var/lib/kubelet/pods/c342fcbd-bae7-4776-9e79-41e49b647daa/volumes"
Nov 03 06:18:54.072561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.072518    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"]
Nov 03 06:18:54.073087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.073001    4528 cpu_manager.go:414] "containerMap: removing container" podUID="907d069a-e8b4-413f-bc7a-bf4b937730c3" containerName="revoke-keys"
Nov 03 06:18:54.073087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.073014    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="907d069a-e8b4-413f-bc7a-bf4b937730c3" containerName="revoke-keys"
Nov 03 06:18:54.073217 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.073117    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="907d069a-e8b4-413f-bc7a-bf4b937730c3" containerName="revoke-keys"
Nov 03 06:18:54.076606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.076586    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.114512 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.114485    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:18:54.131835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.126133    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:18:54.131835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.126192    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:18:54.131835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.126216    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:18:54.142856 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.141977    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"]
Nov 03 06:18:54.155963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.155926    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.156112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.155978    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdqmh\" (UniqueName: \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.156112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.156098    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.257310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.257274    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.257490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.257334    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdqmh\" (UniqueName: \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.257490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.257387    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.258223 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.258189    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.259709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.259682    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.280629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.280602    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdqmh\" (UniqueName: \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\") pod \"maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.392895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.392784    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:54.693406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:54.693376    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"]
Nov 03 06:18:54.695297 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:18:54.695268    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d47a7c2_3330_450b_b340_b2bf0a7f486e.slice/crio-26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95 WatchSource:0}: Error finding container 26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95: Status 404 returned error can't find the container with id 26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95
Nov 03 06:18:55.365418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:55.365380    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf" event={"ID":"5d47a7c2-3330-450b-b340-b2bf0a7f486e","Type":"ContainerStarted","Data":"60fdb4443bc1ef9cc4f89771d770f4f51fb9d817156acd4cb1672d15eda954f4"}
Nov 03 06:18:55.365886 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:55.365434    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf" event={"ID":"5d47a7c2-3330-450b-b340-b2bf0a7f486e","Type":"ContainerStarted","Data":"26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95"}
Nov 03 06:18:55.408423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:55.408342    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf" podStartSLOduration=1.40832577 podStartE2EDuration="1.40832577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:18:55.404254319 +0000 UTC m=+1423.815651399" watchObservedRunningTime="2026-11-03 06:18:55.40832577 +0000 UTC m=+1423.819722851"
Nov 03 06:18:56.373322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:56.373288    4528 generic.go:356] "Generic (PLEG): container finished" podID="5d47a7c2-3330-450b-b340-b2bf0a7f486e" containerID="60fdb4443bc1ef9cc4f89771d770f4f51fb9d817156acd4cb1672d15eda954f4" exitCode=0
Nov 03 06:18:56.373704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:56.373378    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf" event={"ID":"5d47a7c2-3330-450b-b340-b2bf0a7f486e","Type":"ContainerDied","Data":"60fdb4443bc1ef9cc4f89771d770f4f51fb9d817156acd4cb1672d15eda954f4"}
Nov 03 06:18:57.522445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.522421    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:57.594318 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.594285    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\" (UniqueName: \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\") pod \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") "
Nov 03 06:18:57.594498 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.594344    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\") pod \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") "
Nov 03 06:18:57.594569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.594498    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\") pod \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\" (UID: \"5d47a7c2-3330-450b-b340-b2bf0a7f486e\") "
Nov 03 06:18:57.594916 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.594883    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca" pod "5d47a7c2-3330-450b-b340-b2bf0a7f486e" (UID: "5d47a7c2-3330-450b-b340-b2bf0a7f486e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:18:57.596485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.596457    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh" pod "5d47a7c2-3330-450b-b340-b2bf0a7f486e" (UID: "5d47a7c2-3330-450b-b340-b2bf0a7f486e"). InnerVolumeSpecName "kube-api-access-tdqmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:18:57.596485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.596473    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca" pod "5d47a7c2-3330-450b-b340-b2bf0a7f486e" (UID: "5d47a7c2-3330-450b-b340-b2bf0a7f486e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:18:57.695478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.695394    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdqmh\" (UniqueName: \"kubernetes.io/projected/5d47a7c2-3330-450b-b340-b2bf0a7f486e-kube-api-access-tdqmh\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:57.695478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.695422    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:57.695478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:57.695432    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d47a7c2-3330-450b-b340-b2bf0a7f486e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:18:58.392113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:58.392081    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"
Nov 03 06:18:58.392113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:58.392102    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf" event={"ID":"5d47a7c2-3330-450b-b340-b2bf0a7f486e","Type":"ContainerDied","Data":"26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95"}
Nov 03 06:18:58.392332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:58.392161    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c8205c59fdcebfc83839faf14dc88d9c4ef8c6fd312910f93c0c8c3e92ee95"
Nov 03 06:18:58.779629 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:18:58.779547    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:19:04.520257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:04.520210    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:19:04.578524 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:04.578486    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:19:07.791667 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:07.791625    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"]
Nov 03 06:19:07.810781 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:07.810739    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-fcbaef-s9pdf"]
Nov 03 06:19:09.521201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:09.521152    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing" containerID="cri-o://2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b" gracePeriod=25
Nov 03 06:19:09.581596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:09.581549    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing" containerID="cri-o://2442531b4d8d2267325d4b0a501b1ffba83748fe70de537177d71ce305c7b4e4" gracePeriod=25
Nov 03 06:19:09.762059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:09.762010    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d47a7c2-3330-450b-b340-b2bf0a7f486e" path="/var/lib/kubelet/pods/5d47a7c2-3330-450b-b340-b2bf0a7f486e/volumes"
Nov 03 06:19:13.030999 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:13.030964    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"]
Nov 03 06:19:13.041510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:13.041473    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xstgh"]
Nov 03 06:19:13.692443 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:19:13.692406    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:19:13.750458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:13.750423    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3" path="/var/lib/kubelet/pods/bbd2cdbf-b452-4833-be9a-0c7c8e5dd8c3/volumes"
Nov 03 06:19:13.799934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:13.799892    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" podUID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" containerName="main" containerID="cri-o://1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f" gracePeriod=45
Nov 03 06:19:14.053722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.053697    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:19:14.168574 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168539    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.168768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168580    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.168768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168639    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.168768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168708    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\" (UniqueName: \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.168768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168750    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.169067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168877    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:19:14.169067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168883    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\") pod \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\" (UID: \"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281\") "
Nov 03 06:19:14.169067 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.168976    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:19:14.169219 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.169112    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:19:14.169414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.169387    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.169414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.169414    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.169622 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.169428    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.171013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.170990    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:19:14.171118 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.171067    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:19:14.171118 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.171068    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp" pod "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" (UID: "ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281"). InnerVolumeSpecName "kube-api-access-c66dp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:19:14.270510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.270414    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.270510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.270445    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.270510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.270457    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c66dp\" (UniqueName: \"kubernetes.io/projected/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281-kube-api-access-c66dp\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:14.501606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.501571    4528 generic.go:356] "Generic (PLEG): container finished" podID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" containerID="1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f" exitCode=0
Nov 03 06:19:14.501831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.501638    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"
Nov 03 06:19:14.501831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.501664    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" event={"ID":"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281","Type":"ContainerDied","Data":"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"}
Nov 03 06:19:14.501831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.501708    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf" event={"ID":"ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281","Type":"ContainerDied","Data":"17c7aa87e9c956ff98c23e30ddbeef32369e90bef3cb338b6ca34465fdd797ed"}
Nov 03 06:19:14.501831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.501726    4528 scope.go:122] "RemoveContainer" containerID="1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"
Nov 03 06:19:14.513187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.513165    4528 scope.go:122] "RemoveContainer" containerID="1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"
Nov 03 06:19:14.513456 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:19:14.513438    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f\": container with ID starting with 1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f not found: ID does not exist" containerID="1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"
Nov 03 06:19:14.513535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.513471    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f"} err="failed to get container status \"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f\": rpc error: code = NotFound desc = could not find container \"1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f\": container with ID starting with 1aad2edccdb1bf74b2a54cf9c7ebfae4ee9b00b4659828e80edd27a19804a70f not found: ID does not exist"
Nov 03 06:19:14.555167 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.555137    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:19:14.566237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:14.566211    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-fcbaef/simulator-single-main-fcbaef-kserve-7f5c555759-pqhlf"]
Nov 03 06:19:15.181155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:15.181118    4528 scope.go:122] "RemoveContainer" containerID="91445f1a47fcd70790cccd328715799c8d7150bec92bd8b0af315139a5fb663a"
Nov 03 06:19:15.748469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:15.748434    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" path="/var/lib/kubelet/pods/ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281/volumes"
Nov 03 06:19:16.469525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.469486    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"]
Nov 03 06:19:16.480659 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.480623    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d28jd"]
Nov 03 06:19:16.518733 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.518698    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"]
Nov 03 06:19:16.519329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519309    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5d47a7c2-3330-450b-b340-b2bf0a7f486e" containerName="revoke-keys"
Nov 03 06:19:16.519417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519330    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d47a7c2-3330-450b-b340-b2bf0a7f486e" containerName="revoke-keys"
Nov 03 06:19:16.519417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519348    4528 cpu_manager.go:414] "containerMap: removing container" podUID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" containerName="main"
Nov 03 06:19:16.519417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519353    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" containerName="main"
Nov 03 06:19:16.519540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519422    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba88cff6-ffb9-4e7c-9a0b-1b6627dd3281" containerName="main"
Nov 03 06:19:16.519540 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.519439    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d47a7c2-3330-450b-b340-b2bf0a7f486e" containerName="revoke-keys"
Nov 03 06:19:16.524267 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.524249    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.540090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.538700    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:19:16.540090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.538743    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:19:16.540090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.538875    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:19:16.540090 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.538948    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:19:16.541972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.541942    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"]
Nov 03 06:19:16.593216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.593184    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7nrl\" (UniqueName: \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.593391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.593309    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.593391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.593358    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.694269 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.694221    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7nrl\" (UniqueName: \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.694456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.694369    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.694456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.694417    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.695076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.695051    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.696963 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.696936    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.717635 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.717603    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7nrl\" (UniqueName: \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\") pod \"maas-api-revoke-sub-3d4156f1-zvmlx\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:16.838334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:16.838298    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:17.142704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.142677    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"]
Nov 03 06:19:17.143531 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:19:17.143500    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d78c5c_550b_46d2_8916_14d4c6828ffb.slice/crio-581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a WatchSource:0}: Error finding container 581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a: Status 404 returned error can't find the container with id 581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a
Nov 03 06:19:17.523201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.523168    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx" event={"ID":"75d78c5c-550b-46d2-8916-14d4c6828ffb","Type":"ContainerStarted","Data":"8533d0b74fe72218a7e72a84548bfa27c46ac0e2f56763703a507c840ccb9428"}
Nov 03 06:19:17.523580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.523230    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx" event={"ID":"75d78c5c-550b-46d2-8916-14d4c6828ffb","Type":"ContainerStarted","Data":"581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a"}
Nov 03 06:19:17.564322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.564267    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx" podStartSLOduration=1.5642542609999999 podStartE2EDuration="1.564254261s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:19:17.56265689 +0000 UTC m=+1445.974053973" watchObservedRunningTime="2026-11-03 06:19:17.564254261 +0000 UTC m=+1445.975651369"
Nov 03 06:19:17.747961 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.747884    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dda55bf-6239-4a99-b133-959eb89a0ffb" path="/var/lib/kubelet/pods/2dda55bf-6239-4a99-b133-959eb89a0ffb/volumes"
Nov 03 06:19:17.966448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.966403    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:19:17.966448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:17.966437    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:19:18.531075 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:18.530975    4528 generic.go:356] "Generic (PLEG): container finished" podID="75d78c5c-550b-46d2-8916-14d4c6828ffb" containerID="8533d0b74fe72218a7e72a84548bfa27c46ac0e2f56763703a507c840ccb9428" exitCode=0
Nov 03 06:19:18.531437 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:18.531065    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx" event={"ID":"75d78c5c-550b-46d2-8916-14d4c6828ffb","Type":"ContainerDied","Data":"8533d0b74fe72218a7e72a84548bfa27c46ac0e2f56763703a507c840ccb9428"}
Nov 03 06:19:19.679844 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.679819    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:19.836657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.836623    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\") pod \"75d78c5c-550b-46d2-8916-14d4c6828ffb\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") "
Nov 03 06:19:19.836871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.836679    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\") pod \"75d78c5c-550b-46d2-8916-14d4c6828ffb\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") "
Nov 03 06:19:19.836871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.836817    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\" (UniqueName: \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\") pod \"75d78c5c-550b-46d2-8916-14d4c6828ffb\" (UID: \"75d78c5c-550b-46d2-8916-14d4c6828ffb\") "
Nov 03 06:19:19.837134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.837096    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca" pod "75d78c5c-550b-46d2-8916-14d4c6828ffb" (UID: "75d78c5c-550b-46d2-8916-14d4c6828ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:19:19.837296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.837275    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:19.839063 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.839040    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl" pod "75d78c5c-550b-46d2-8916-14d4c6828ffb" (UID: "75d78c5c-550b-46d2-8916-14d4c6828ffb"). InnerVolumeSpecName "kube-api-access-v7nrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:19:19.839162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.839114    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca" pod "75d78c5c-550b-46d2-8916-14d4c6828ffb" (UID: "75d78c5c-550b-46d2-8916-14d4c6828ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:19:19.938144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.938104    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7nrl\" (UniqueName: \"kubernetes.io/projected/75d78c5c-550b-46d2-8916-14d4c6828ffb-kube-api-access-v7nrl\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:19.938144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:19.938136    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75d78c5c-550b-46d2-8916-14d4c6828ffb-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:20.545218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:20.545188    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"
Nov 03 06:19:20.545218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:20.545199    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx" event={"ID":"75d78c5c-550b-46d2-8916-14d4c6828ffb","Type":"ContainerDied","Data":"581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a"}
Nov 03 06:19:20.545431 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:20.545272    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="581c5e45626b3fa8c214536860278910e296a322f88a3baa7f3cdc373557067a"
Nov 03 06:19:24.029596 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:24.029536    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"]
Nov 03 06:19:24.041096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:24.041061    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-qw4cc"]
Nov 03 06:19:25.747897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:25.747863    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4406891-25a7-4b89-bfd6-d9823855221a" path="/var/lib/kubelet/pods/e4406891-25a7-4b89-bfd6-d9823855221a/volumes"
Nov 03 06:19:27.966220 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:27.966179    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:19:27.966611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:27.966179    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 06:19:30.460221 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.459810    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:19:30.488257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.487522    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\") pod \"b391bc86-1250-4b19-b174-12a02dabd774\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") "
Nov 03 06:19:30.488257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.487608    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\" (UniqueName: \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\") pod \"b391bc86-1250-4b19-b174-12a02dabd774\" (UID: \"b391bc86-1250-4b19-b174-12a02dabd774\") "
Nov 03 06:19:30.488962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.488915    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume" pod "b391bc86-1250-4b19-b174-12a02dabd774" (UID: "b391bc86-1250-4b19-b174-12a02dabd774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:19:30.588978 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.588633    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b391bc86-1250-4b19-b174-12a02dabd774-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:30.627962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.627901    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n" pod "b391bc86-1250-4b19-b174-12a02dabd774" (UID: "b391bc86-1250-4b19-b174-12a02dabd774"). InnerVolumeSpecName "kube-api-access-55h6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:19:30.689879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.689829    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-55h6n\" (UniqueName: \"kubernetes.io/projected/b391bc86-1250-4b19-b174-12a02dabd774-kube-api-access-55h6n\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:30.730445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.704439    4528 generic.go:356] "Generic (PLEG): container finished" podID="b391bc86-1250-4b19-b174-12a02dabd774" containerID="2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b" exitCode=0
Nov 03 06:19:30.730445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.704657    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" event={"ID":"b391bc86-1250-4b19-b174-12a02dabd774","Type":"ContainerDied","Data":"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"}
Nov 03 06:19:30.730445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.704715    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q" event={"ID":"b391bc86-1250-4b19-b174-12a02dabd774","Type":"ContainerDied","Data":"271cbfd4f8c59550a7b259bac2c92ee832094a11a8d338cb29594481522fca5f"}
Nov 03 06:19:30.730445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.704747    4528 scope.go:122] "RemoveContainer" containerID="2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"
Nov 03 06:19:30.730445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.704949    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"
Nov 03 06:19:30.771245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.771120    4528 generic.go:356] "Generic (PLEG): container finished" podID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerID="2442531b4d8d2267325d4b0a501b1ffba83748fe70de537177d71ce305c7b4e4" exitCode=0
Nov 03 06:19:30.771245 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.771175    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" event={"ID":"919e6e4e-fb34-4c6a-89c8-201fabda5bf1","Type":"ContainerDied","Data":"2442531b4d8d2267325d4b0a501b1ffba83748fe70de537177d71ce305c7b4e4"}
Nov 03 06:19:30.853929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.853640    4528 scope.go:122] "RemoveContainer" containerID="2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"
Nov 03 06:19:30.857954 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:19:30.857872    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b\": container with ID starting with 2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b not found: ID does not exist" containerID="2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"
Nov 03 06:19:30.857954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.857919    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b"} err="failed to get container status \"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b\": rpc error: code = NotFound desc = could not find container \"2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b\": container with ID starting with 2889be4a36bc1fa854ea7ebaf077a175494d0413333d4372e65a1e66241c283b not found: ID does not exist"
Nov 03 06:19:30.936823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.936532    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:19:30.938991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.938966    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:19:30.965819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:30.963022    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-fcbaef-54bdc48d68tr6q"]
Nov 03 06:19:31.042442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.042405    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\") pod \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") "
Nov 03 06:19:31.045909 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.045871    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\" (UniqueName: \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\") pod \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\" (UID: \"919e6e4e-fb34-4c6a-89c8-201fabda5bf1\") "
Nov 03 06:19:31.065234 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.065197    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume" pod "919e6e4e-fb34-4c6a-89c8-201fabda5bf1" (UID: "919e6e4e-fb34-4c6a-89c8-201fabda5bf1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:19:31.067376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.067343    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw" pod "919e6e4e-fb34-4c6a-89c8-201fabda5bf1" (UID: "919e6e4e-fb34-4c6a-89c8-201fabda5bf1"). InnerVolumeSpecName "kube-api-access-7kwlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:19:31.152876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.152762    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:31.152876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.152816    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kwlw\" (UniqueName: \"kubernetes.io/projected/919e6e4e-fb34-4c6a-89c8-201fabda5bf1-kube-api-access-7kwlw\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:31.747852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.747819    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b391bc86-1250-4b19-b174-12a02dabd774" path="/var/lib/kubelet/pods/b391bc86-1250-4b19-b174-12a02dabd774/volumes"
Nov 03 06:19:31.780034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.779997    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7" event={"ID":"919e6e4e-fb34-4c6a-89c8-201fabda5bf1","Type":"ContainerDied","Data":"d8e1d0ef0a10c62e8d3554be8e2da8b55c79e24c1b3a1d8c41ef23c4abf0f422"}
Nov 03 06:19:31.780251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.780076    4528 scope.go:122] "RemoveContainer" containerID="2442531b4d8d2267325d4b0a501b1ffba83748fe70de537177d71ce305c7b4e4"
Nov 03 06:19:31.780251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.780013    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"
Nov 03 06:19:31.833897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.833864    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:19:31.851823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:31.847493    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-fcbaef-676c5f6qg7"]
Nov 03 06:19:33.748316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:33.748277    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" path="/var/lib/kubelet/pods/919e6e4e-fb34-4c6a-89c8-201fabda5bf1/volumes"
Nov 03 06:19:40.755383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:40.755346    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 06:19:40.756032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:40.755564    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" podUID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" containerName="manager" containerID="cri-o://635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef" gracePeriod=10
Nov 03 06:19:41.032035 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.032002    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 06:19:41.148553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.148519    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\" (UniqueName: \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\") pod \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") "
Nov 03 06:19:41.148719 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.148597    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\") pod \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\" (UID: \"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4\") "
Nov 03 06:19:41.149084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.149055    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume" pod "c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" (UID: "c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:19:41.150613 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.150594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl" pod "c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" (UID: "c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4"). InnerVolumeSpecName "kube-api-access-5t5nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:19:41.249923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.249881    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5t5nl\" (UniqueName: \"kubernetes.io/projected/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-kube-api-access-5t5nl\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:41.249923 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.249914    4528 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4-extensions-socket-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:19:41.864304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.864254    4528 generic.go:356] "Generic (PLEG): container finished" podID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" containerID="635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef" exitCode=0
Nov 03 06:19:41.864722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.864324    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"
Nov 03 06:19:41.864722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.864344    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" event={"ID":"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4","Type":"ContainerDied","Data":"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"}
Nov 03 06:19:41.864722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.864402    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh" event={"ID":"c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4","Type":"ContainerDied","Data":"4c22213cffbda40c89383a9072d90c484881935485167153fc6b3c03117ca5e4"}
Nov 03 06:19:41.864722 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.864425    4528 scope.go:122] "RemoveContainer" containerID="635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"
Nov 03 06:19:41.876260 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.876237    4528 scope.go:122] "RemoveContainer" containerID="635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"
Nov 03 06:19:41.876623 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:19:41.876597    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef\": container with ID starting with 635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef not found: ID does not exist" containerID="635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"
Nov 03 06:19:41.876699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.876633    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef"} err="failed to get container status \"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef\": rpc error: code = NotFound desc = could not find container \"635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef\": container with ID starting with 635ebb74b5db2afec9c6fff4f705601ece1b2b61cb77898bb669300968e587ef not found: ID does not exist"
Nov 03 06:19:41.927788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.927753    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 06:19:41.939756 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:41.939716    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-t49wh"]
Nov 03 06:19:43.692191 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:19:43.692154    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:19:43.747492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:43.747461    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" path="/var/lib/kubelet/pods/c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4/volumes"
Nov 03 06:19:48.029630 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:48.029590    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"]
Nov 03 06:19:48.040598 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:48.040566    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8vvxn"]
Nov 03 06:19:49.747451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:19:49.747415    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5" path="/var/lib/kubelet/pods/65ebe2eb-31e8-4b42-b39d-7f6c7de1eaa5/volumes"
Nov 03 06:20:13.692301 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:20:13.692256    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:20:15.219743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:15.219705    4528 scope.go:122] "RemoveContainer" containerID="7985c8a43ae3086bf2bd8fb8773c7286d999e9e9c7a8afddb71673095637b2cc"
Nov 03 06:20:15.229230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:15.229201    4528 scope.go:122] "RemoveContainer" containerID="8e944eba9cf7dd790d6bd43a123e0e2276e695c58d399ec29f79ace9fe0548ce"
Nov 03 06:20:15.238096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:15.238074    4528 scope.go:122] "RemoveContainer" containerID="5d575dfea9aad1e47604d5fcc42d244064dccaa317777e3f1730d39b5a7ccfa4"
Nov 03 06:20:33.027855 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:33.027818    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"]
Nov 03 06:20:33.038967 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:33.038941    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-xxjhf"]
Nov 03 06:20:33.746970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:33.746937    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcc6b768-fd1a-4830-b8db-eaa24e294102" path="/var/lib/kubelet/pods/dcc6b768-fd1a-4830-b8db-eaa24e294102/volumes"
Nov 03 06:20:43.692463 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:20:43.692423    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:20:44.001683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.001599    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"]
Nov 03 06:20:44.002051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002038    4528 cpu_manager.go:414] "containerMap: removing container" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002052    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002063    4528 cpu_manager.go:414] "containerMap: removing container" podUID="75d78c5c-550b-46d2-8916-14d4c6828ffb" containerName="revoke-keys"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002068    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75d78c5c-550b-46d2-8916-14d4c6828ffb" containerName="revoke-keys"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002087    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" containerName="manager"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002092    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" containerName="manager"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002108    4528 cpu_manager.go:414] "containerMap: removing container" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing"
Nov 03 06:20:44.002116 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002115    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing"
Nov 03 06:20:44.002364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002183    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="b391bc86-1250-4b19-b174-12a02dabd774" containerName="payload-processing"
Nov 03 06:20:44.002364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002192    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="75d78c5c-550b-46d2-8916-14d4c6828ffb" containerName="revoke-keys"
Nov 03 06:20:44.002364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002201    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c109b7c3-aa29-436e-9e3d-6c3e2e8bcdb4" containerName="manager"
Nov 03 06:20:44.002364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.002209    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="919e6e4e-fb34-4c6a-89c8-201fabda5bf1" containerName="payload-pre-processing"
Nov 03 06:20:44.005727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.005710    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.020718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.020689    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"]
Nov 03 06:20:44.029908 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.029871    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-lw66s\""
Nov 03 06:20:44.176028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.175991    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/dd057882-81dc-4bd5-b249-4ea76ecbe396-kube-api-access-nxplt\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.176216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.176048    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd057882-81dc-4bd5-b249-4ea76ecbe396-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.276863 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.276758    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/dd057882-81dc-4bd5-b249-4ea76ecbe396-kube-api-access-nxplt\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.277011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.276863    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd057882-81dc-4bd5-b249-4ea76ecbe396-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.277240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.277221    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dd057882-81dc-4bd5-b249-4ea76ecbe396-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.301353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.301319    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxplt\" (UniqueName: \"kubernetes.io/projected/dd057882-81dc-4bd5-b249-4ea76ecbe396-kube-api-access-nxplt\") pod \"kuadrant-operator-controller-manager-678d87766f-g6r8h\" (UID: \"dd057882-81dc-4bd5-b249-4ea76ecbe396\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.319476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.319434    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:44.655371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:44.655337    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"]
Nov 03 06:20:44.656369 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:20:44.656343    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd057882_81dc_4bd5_b249_4ea76ecbe396.slice/crio-3cdf5abc59094e2708aea503becbed8736eb2fbef1220aab8867bbb164d3c776 WatchSource:0}: Error finding container 3cdf5abc59094e2708aea503becbed8736eb2fbef1220aab8867bbb164d3c776: Status 404 returned error can't find the container with id 3cdf5abc59094e2708aea503becbed8736eb2fbef1220aab8867bbb164d3c776
Nov 03 06:20:45.264429 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:45.264391    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h" event={"ID":"dd057882-81dc-4bd5-b249-4ea76ecbe396","Type":"ContainerStarted","Data":"91e858df3fbe75267e499009d495cba5c16cc59cb1a8de36489799ba0d6a7afe"}
Nov 03 06:20:45.264965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:45.264481    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h" event={"ID":"dd057882-81dc-4bd5-b249-4ea76ecbe396","Type":"ContainerStarted","Data":"3cdf5abc59094e2708aea503becbed8736eb2fbef1220aab8867bbb164d3c776"}
Nov 03 06:20:45.264965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:45.264541    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:20:45.331490 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:45.331433    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h" podStartSLOduration=2.3314175759999998 podStartE2EDuration="2.331417576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:20:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:20:45.318519794 +0000 UTC m=+1533.729916871" watchObservedRunningTime="2026-11-03 06:20:45.331417576 +0000 UTC m=+1533.742814656"
Nov 03 06:20:52.032835 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:52.032785    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"]
Nov 03 06:20:52.044192 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:52.044160    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-glg9r"]
Nov 03 06:20:53.747272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:53.747238    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffb9567-2a57-4ec7-acb6-763d2b489911" path="/var/lib/kubelet/pods/6ffb9567-2a57-4ec7-acb6-763d2b489911/volumes"
Nov 03 06:20:56.273784 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:20:56.273750    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-g6r8h"
Nov 03 06:21:05.960455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.960394    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"]
Nov 03 06:21:05.965230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.965197    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:05.982017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.981972    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"]
Nov 03 06:21:05.992494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.992464    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:21:05.992678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.992558    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:21:05.992678 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.992644    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:21:05.993990 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:05.993960    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:21:06.063857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.063819    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-625s2\" (UniqueName: \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.064073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.063912    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.064073 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.064041    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.165133 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.165090    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.165328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.165218    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-625s2\" (UniqueName: \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.165328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.165248    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.165760 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.165734    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.167603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.167581    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.188500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.188473    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-625s2\" (UniqueName: \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\") pod \"maas-api-revoke-sub-d576e68b-rwzl6\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.286536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.286460    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:06.583438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:06.583400    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"]
Nov 03 06:21:06.583930 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:21:06.583904    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4f6270_1d1f_46ae_82a8_d5d1c3426743.slice/crio-d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6 WatchSource:0}: Error finding container d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6: Status 404 returned error can't find the container with id d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6
Nov 03 06:21:07.476289 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:07.476244    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6" event={"ID":"0f4f6270-1d1f-46ae-82a8-d5d1c3426743","Type":"ContainerStarted","Data":"446b4670b158c920a884aa8bfa57fd0b80c65261bf95a3a6314db76da3bcf364"}
Nov 03 06:21:07.476715 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:07.476319    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6" event={"ID":"0f4f6270-1d1f-46ae-82a8-d5d1c3426743","Type":"ContainerStarted","Data":"d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6"}
Nov 03 06:21:07.518113 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:07.518064    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6" podStartSLOduration=2.518049333 podStartE2EDuration="2.518049333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:07.517978702 +0000 UTC m=+1555.929375782" watchObservedRunningTime="2026-11-03 06:21:07.518049333 +0000 UTC m=+1555.929446410"
Nov 03 06:21:08.490989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:08.490955    4528 generic.go:356] "Generic (PLEG): container finished" podID="0f4f6270-1d1f-46ae-82a8-d5d1c3426743" containerID="446b4670b158c920a884aa8bfa57fd0b80c65261bf95a3a6314db76da3bcf364" exitCode=0
Nov 03 06:21:08.491374 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:08.491042    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6" event={"ID":"0f4f6270-1d1f-46ae-82a8-d5d1c3426743","Type":"ContainerDied","Data":"446b4670b158c920a884aa8bfa57fd0b80c65261bf95a3a6314db76da3bcf364"}
Nov 03 06:21:09.635676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.635646    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:09.699745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.699708    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\") pod \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") "
Nov 03 06:21:09.699944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.699772    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\") pod \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") "
Nov 03 06:21:09.699944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.699829    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\" (UniqueName: \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\") pod \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\" (UID: \"0f4f6270-1d1f-46ae-82a8-d5d1c3426743\") "
Nov 03 06:21:09.700195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.700167    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca" pod "0f4f6270-1d1f-46ae-82a8-d5d1c3426743" (UID: "0f4f6270-1d1f-46ae-82a8-d5d1c3426743"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:21:09.702051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.702024    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca" pod "0f4f6270-1d1f-46ae-82a8-d5d1c3426743" (UID: "0f4f6270-1d1f-46ae-82a8-d5d1c3426743"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:21:09.702051 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.702041    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2" pod "0f4f6270-1d1f-46ae-82a8-d5d1c3426743" (UID: "0f4f6270-1d1f-46ae-82a8-d5d1c3426743"). InnerVolumeSpecName "kube-api-access-625s2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:21:09.801161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.801126    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-625s2\" (UniqueName: \"kubernetes.io/projected/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-kube-api-access-625s2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:09.801161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.801156    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:09.801161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:09.801165    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f4f6270-1d1f-46ae-82a8-d5d1c3426743-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:10.505338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:10.505304    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6" event={"ID":"0f4f6270-1d1f-46ae-82a8-d5d1c3426743","Type":"ContainerDied","Data":"d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6"}
Nov 03 06:21:10.505529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:10.505359    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2a2e3c730ebfe506e4f1183ddf5ecfca674dd709d9bd4c1d65bd2fc1f9f16b6"
Nov 03 06:21:10.505529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:10.505313    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"
Nov 03 06:21:13.692620 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:21:13.692585    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:21:15.291632 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:15.291598    4528 scope.go:122] "RemoveContainer" containerID="27ee58a7506ac6872464d1d76ccba2b77b8612ee309b74ccc7e4f0b5cf17d495"
Nov 03 06:21:15.301150 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:15.301128    4528 scope.go:122] "RemoveContainer" containerID="3db382fdcd354876a68c9ea802f7d57970eb4c6817aa96294041bd02dd0b6bf0"
Nov 03 06:21:22.703829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:22.702911    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:21:22.703829 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:22.702991    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:21:22.877817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:22.866908    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-q2zk7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:21:22.877817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:22.866994    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-q2zk7" podUID="138857e2-d465-4298-a8af-7ebfbab95b86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:21:27.448913 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.448834    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"]
Nov 03 06:21:27.449330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.449312    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0f4f6270-1d1f-46ae-82a8-d5d1c3426743" containerName="revoke-keys"
Nov 03 06:21:27.449386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.449333    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f4f6270-1d1f-46ae-82a8-d5d1c3426743" containerName="revoke-keys"
Nov 03 06:21:27.449460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.449449    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f4f6270-1d1f-46ae-82a8-d5d1c3426743" containerName="revoke-keys"
Nov 03 06:21:27.454707 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.454686    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.471648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.471621    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"]
Nov 03 06:21:27.472201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.472182    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.472316 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.472237    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9rtb\" (UniqueName: \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.472385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.472366    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.479494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.479461    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:21:27.479609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.479478    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:21:27.479609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.479517    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:21:27.480854 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.480806    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:21:27.574099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.573943    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.574099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.574032    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.574099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.574109    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9rtb\" (UniqueName: \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.574757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.574731    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.576504 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.576477    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.597285 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.597254    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9rtb\" (UniqueName: \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\") pod \"maas-api-revoke-sub-d2bc93bc-pd686\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:27.768507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:27.768409    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:28.066895 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:28.066868    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"]
Nov 03 06:21:28.072582 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:21:28.072510    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dbada2d_7933_46a8_ae3e_87e6355f8eb8.slice/crio-3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f WatchSource:0}: Error finding container 3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f: Status 404 returned error can't find the container with id 3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f
Nov 03 06:21:28.653076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:28.652977    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686" event={"ID":"4dbada2d-7933-46a8-ae3e-87e6355f8eb8","Type":"ContainerStarted","Data":"95b4f2a4940e26ab01088ce8c889afc2499b577cf6790008b96012de943d59d5"}
Nov 03 06:21:28.653076 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:28.653061    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686" event={"ID":"4dbada2d-7933-46a8-ae3e-87e6355f8eb8","Type":"ContainerStarted","Data":"3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f"}
Nov 03 06:21:28.695643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:28.695579    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686" podStartSLOduration=1.695564523 podStartE2EDuration="1.695564523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:28.692857033 +0000 UTC m=+1577.104254111" watchObservedRunningTime="2026-11-03 06:21:28.695564523 +0000 UTC m=+1577.106961600"
Nov 03 06:21:29.660658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:29.660623    4528 generic.go:356] "Generic (PLEG): container finished" podID="4dbada2d-7933-46a8-ae3e-87e6355f8eb8" containerID="95b4f2a4940e26ab01088ce8c889afc2499b577cf6790008b96012de943d59d5" exitCode=0
Nov 03 06:21:29.661335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:29.660710    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686" event={"ID":"4dbada2d-7933-46a8-ae3e-87e6355f8eb8","Type":"ContainerDied","Data":"95b4f2a4940e26ab01088ce8c889afc2499b577cf6790008b96012de943d59d5"}
Nov 03 06:21:30.801547 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.801521    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:30.915218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.915184    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\" (UniqueName: \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\") pod \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") "
Nov 03 06:21:30.915419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.915347    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\") pod \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") "
Nov 03 06:21:30.915419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.915389    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\") pod \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\" (UID: \"4dbada2d-7933-46a8-ae3e-87e6355f8eb8\") "
Nov 03 06:21:30.915780 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.915746    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca" pod "4dbada2d-7933-46a8-ae3e-87e6355f8eb8" (UID: "4dbada2d-7933-46a8-ae3e-87e6355f8eb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:21:30.917354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.917333    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb" pod "4dbada2d-7933-46a8-ae3e-87e6355f8eb8" (UID: "4dbada2d-7933-46a8-ae3e-87e6355f8eb8"). InnerVolumeSpecName "kube-api-access-c9rtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:21:30.917489 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:30.917472    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca" pod "4dbada2d-7933-46a8-ae3e-87e6355f8eb8" (UID: "4dbada2d-7933-46a8-ae3e-87e6355f8eb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:21:31.016738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.016705    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:31.016738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.016734    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:31.016738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.016744    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9rtb\" (UniqueName: \"kubernetes.io/projected/4dbada2d-7933-46a8-ae3e-87e6355f8eb8-kube-api-access-c9rtb\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:21:31.675192 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.675155    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686" event={"ID":"4dbada2d-7933-46a8-ae3e-87e6355f8eb8","Type":"ContainerDied","Data":"3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f"}
Nov 03 06:21:31.675192 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.675189    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"
Nov 03 06:21:31.675426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:31.675210    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3861ae281be1e6d9737f9e5c23e4feac054c0158851b26d1adfeaac7603ad86f"
Nov 03 06:21:43.692355 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:21:43.692309    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:21:45.620044 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.620006    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:21:45.620493 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.620479    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4dbada2d-7933-46a8-ae3e-87e6355f8eb8" containerName="revoke-keys"
Nov 03 06:21:45.620550 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.620494    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dbada2d-7933-46a8-ae3e-87e6355f8eb8" containerName="revoke-keys"
Nov 03 06:21:45.620587 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.620578    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dbada2d-7933-46a8-ae3e-87e6355f8eb8" containerName="revoke-keys"
Nov 03 06:21:45.622954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.622934    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.643068 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.643035    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:21:45.650452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.650420    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-7c2bc6\""
Nov 03 06:21:45.650643 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.650625    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-50c35a86-dockercfg-99l78\""
Nov 03 06:21:45.685173 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.685149    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:21:45.687935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.687920    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.714608 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.714584    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:21:45.776608 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.776554    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.776919 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.776888    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.777104 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.777086    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df4ng\" (UniqueName: \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.777187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.777170    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgc6\" (UniqueName: \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.878096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.878013    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df4ng\" (UniqueName: \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.878274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.878183    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgc6\" (UniqueName: \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.878274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.878242    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.878401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.878318    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.879001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.878977    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.879125 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.879041    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.901056 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.901024    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df4ng\" (UniqueName: \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\") pod \"payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:45.901166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.901085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgc6\" (UniqueName: \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\") pod \"payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:45.935720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:45.935692    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:46.000120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.000081    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:46.235516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.235483    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:21:46.236095 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:21:46.236068    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c390c2a_0933_4d05_ae93_0a1d0f005c45.slice/crio-db8e7d789ec733b939a615bdcc1e231afaf45c8e41857d418635a7c498f1bcd5 WatchSource:0}: Error finding container db8e7d789ec733b939a615bdcc1e231afaf45c8e41857d418635a7c498f1bcd5: Status 404 returned error can't find the container with id db8e7d789ec733b939a615bdcc1e231afaf45c8e41857d418635a7c498f1bcd5
Nov 03 06:21:46.309456 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:21:46.309429    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod340d1b46_b755_442e_b348_d4c9dd86a0c9.slice/crio-393dcb154edc06800b9d213793faf6075b21d6ee89dcdf2c2321d410831f9d5e WatchSource:0}: Error finding container 393dcb154edc06800b9d213793faf6075b21d6ee89dcdf2c2321d410831f9d5e: Status 404 returned error can't find the container with id 393dcb154edc06800b9d213793faf6075b21d6ee89dcdf2c2321d410831f9d5e
Nov 03 06:21:46.309820 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.309773    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:21:46.809868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.809827    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" event={"ID":"8c390c2a-0933-4d05-ae93-0a1d0f005c45","Type":"ContainerStarted","Data":"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"}
Nov 03 06:21:46.810313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.809895    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:21:46.810313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.809921    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" event={"ID":"8c390c2a-0933-4d05-ae93-0a1d0f005c45","Type":"ContainerStarted","Data":"db8e7d789ec733b939a615bdcc1e231afaf45c8e41857d418635a7c498f1bcd5"}
Nov 03 06:21:46.811799 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.811765    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" event={"ID":"340d1b46-b755-442e-b348-d4c9dd86a0c9","Type":"ContainerStarted","Data":"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"}
Nov 03 06:21:46.811917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.811820    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" event={"ID":"340d1b46-b755-442e-b348-d4c9dd86a0c9","Type":"ContainerStarted","Data":"393dcb154edc06800b9d213793faf6075b21d6ee89dcdf2c2321d410831f9d5e"}
Nov 03 06:21:46.811917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.811840    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:46.853264 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.853202    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" podStartSLOduration=1.85318162 podStartE2EDuration="1.85318162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:46.849133416 +0000 UTC m=+1595.260530496" watchObservedRunningTime="2026-11-03 06:21:46.85318162 +0000 UTC m=+1595.264578700"
Nov 03 06:21:46.889639 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:46.889575    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" podStartSLOduration=1.889560594 podStartE2EDuration="1.889560594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:21:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:21:46.888635436 +0000 UTC m=+1595.300032516" watchObservedRunningTime="2026-11-03 06:21:46.889560594 +0000 UTC m=+1595.300957672"
Nov 03 06:21:57.825734 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:57.825700    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:21:57.826226 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:21:57.825776    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:22:01.315273 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.315237    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:22:01.319224 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.319205    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.334553 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.334518    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7c2bc6\"/\"openshift-service-ca.crt\""
Nov 03 06:22:01.346040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346007    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-7c2bc6\"/\"kube-root-ca.crt\""
Nov 03 06:22:01.346232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346150    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7c2bc6\"/\"default-dockercfg-s7r8f\""
Nov 03 06:22:01.346322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346281    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-7c2bc6\"/\"simulator-single-main-7c2bc6-kserve-self-signed-certs\""
Nov 03 06:22:01.346899 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346876    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.346998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346925    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.346998 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.346945    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.347115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.347079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.347178 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.347126    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8c9w\" (UniqueName: \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.347254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.347238    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.349626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.349605    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:22:01.447803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.447768    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.447868    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.447904    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.447922    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.447978    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448013 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.448013    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8c9w\" (UniqueName: \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448383 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.448363    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.448424    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.448491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.448467    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.450338 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.450307    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.450441 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.450383    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.471428 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.471394    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8c9w\" (UniqueName: \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\") pod \"simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") " pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.632031 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.631947    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:01.942944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:01.942912    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:22:01.943494 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:22:01.943470    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9971edd1_8208_4ea2_a118_e33ca6109574.slice/crio-3b6db79c92616fca8e3c3233f1896ef93690f079fcf100bcc6c3c18aa83178b1 WatchSource:0}: Error finding container 3b6db79c92616fca8e3c3233f1896ef93690f079fcf100bcc6c3c18aa83178b1: Status 404 returned error can't find the container with id 3b6db79c92616fca8e3c3233f1896ef93690f079fcf100bcc6c3c18aa83178b1
Nov 03 06:22:02.930313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:02.930278    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" event={"ID":"9971edd1-8208-4ea2-a118-e33ca6109574","Type":"ContainerStarted","Data":"216abf44105e93a825e1f701970b3bae21492476875719fa2e320157dce3fda7"}
Nov 03 06:22:02.930313 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:02.930321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" event={"ID":"9971edd1-8208-4ea2-a118-e33ca6109574","Type":"ContainerStarted","Data":"3b6db79c92616fca8e3c3233f1896ef93690f079fcf100bcc6c3c18aa83178b1"}
Nov 03 06:22:02.975823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:02.975752    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" podStartSLOduration=1.975739235 podStartE2EDuration="1.975739235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:22:02.972354928 +0000 UTC m=+1611.383752031" watchObservedRunningTime="2026-11-03 06:22:02.975739235 +0000 UTC m=+1611.387136311"
Nov 03 06:22:03.027859 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:03.027816    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"]
Nov 03 06:22:03.031435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:03.031408    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-dgxg2"]
Nov 03 06:22:03.632500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:03.632470    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:03.748473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:03.748436    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da394fca-7901-42cc-82bc-e360e6c2efa3" path="/var/lib/kubelet/pods/da394fca-7901-42cc-82bc-e360e6c2efa3/volumes"
Nov 03 06:22:11.633072 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:11.633030    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:11.645656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:11.645619    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:12.010531 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:12.010455    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:22:13.692356 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:22:13.692315    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:22:14.041544 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:14.041459    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"]
Nov 03 06:22:14.052633 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:14.052603    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-s29vl"]
Nov 03 06:22:15.328776 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:15.328745    4528 scope.go:122] "RemoveContainer" containerID="3a4dad2b95888b2140481a375ae4e91dd59050940caab029ce2cd7f885d58767"
Nov 03 06:22:15.338876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:15.338856    4528 scope.go:122] "RemoveContainer" containerID="f6290d82eb991c0ba7fac69c3b7bc7167d3adb5c53c591cbe2f41670ea6e8ad4"
Nov 03 06:22:15.347343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:15.347318    4528 scope.go:122] "RemoveContainer" containerID="15167bff4564fc610295e4134629dec685a596ea5e50237651873a2ac1cfa06f"
Nov 03 06:22:15.755447 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:15.755354    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7999fed9-5e74-4518-9c32-c23758d989a5" path="/var/lib/kubelet/pods/7999fed9-5e74-4518-9c32-c23758d989a5/volumes"
Nov 03 06:22:33.292216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.292169    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"]
Nov 03 06:22:33.296204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.296177    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.310636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.310581    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:22:33.310816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.310727    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:22:33.310816 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.310744    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:22:33.310938 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.310822    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:22:33.314349 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.314315    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"]
Nov 03 06:22:33.386164 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.386129    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5qbd\" (UniqueName: \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.386350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.386188    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.386350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.386279    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.487488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.487448    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.487717 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.487564    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5qbd\" (UniqueName: \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.487717 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.487605    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.488184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.488159    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.489943 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.489920    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.510740 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.510708    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5qbd\" (UniqueName: \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\") pod \"maas-api-revoke-sub-3a651a15-gcbhm\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.609582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.609544    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:33.914542 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:22:33.914509    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e3a11a5_ece0_473f_9191_0e25881b179f.slice/crio-bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d WatchSource:0}: Error finding container bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d: Status 404 returned error can't find the container with id bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d
Nov 03 06:22:33.915210 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:33.915148    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"]
Nov 03 06:22:34.144745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:34.144709    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm" event={"ID":"2e3a11a5-ece0-473f-9191-0e25881b179f","Type":"ContainerStarted","Data":"bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d"}
Nov 03 06:22:35.153254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:35.153218    4528 generic.go:356] "Generic (PLEG): container finished" podID="2e3a11a5-ece0-473f-9191-0e25881b179f" containerID="9b1da406e1a27219f62f29b4483752d3a29134c9a8c081cc41a938460a818c9f" exitCode=0
Nov 03 06:22:35.153691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:35.153328    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm" event={"ID":"2e3a11a5-ece0-473f-9191-0e25881b179f","Type":"ContainerDied","Data":"9b1da406e1a27219f62f29b4483752d3a29134c9a8c081cc41a938460a818c9f"}
Nov 03 06:22:36.302637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.302611    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:36.418945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.418848    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\" (UniqueName: \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\") pod \"2e3a11a5-ece0-473f-9191-0e25881b179f\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") "
Nov 03 06:22:36.418945 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.418899    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\") pod \"2e3a11a5-ece0-473f-9191-0e25881b179f\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") "
Nov 03 06:22:36.419172 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.419043    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\") pod \"2e3a11a5-ece0-473f-9191-0e25881b179f\" (UID: \"2e3a11a5-ece0-473f-9191-0e25881b179f\") "
Nov 03 06:22:36.419457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.419427    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca" pod "2e3a11a5-ece0-473f-9191-0e25881b179f" (UID: "2e3a11a5-ece0-473f-9191-0e25881b179f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:22:36.421146 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.421120    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca" pod "2e3a11a5-ece0-473f-9191-0e25881b179f" (UID: "2e3a11a5-ece0-473f-9191-0e25881b179f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:22:36.421146 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.421131    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd" pod "2e3a11a5-ece0-473f-9191-0e25881b179f" (UID: "2e3a11a5-ece0-473f-9191-0e25881b179f"). InnerVolumeSpecName "kube-api-access-c5qbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:22:36.520690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.520648    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:36.520690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.520688    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e3a11a5-ece0-473f-9191-0e25881b179f-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:36.520690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:36.520705    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5qbd\" (UniqueName: \"kubernetes.io/projected/2e3a11a5-ece0-473f-9191-0e25881b179f-kube-api-access-c5qbd\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:37.168304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:37.168272    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"
Nov 03 06:22:37.168552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:37.168318    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm" event={"ID":"2e3a11a5-ece0-473f-9191-0e25881b179f","Type":"ContainerDied","Data":"bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d"}
Nov 03 06:22:37.168552 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:37.168370    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf24b434fd910b4321bda3cac56241b5f40e7d659f1b06aafe40b162860cf93d"
Nov 03 06:22:43.693441 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:22:43.693398    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:22:44.423768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.423717    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"]
Nov 03 06:22:44.424522 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.424496    4528 cpu_manager.go:414] "containerMap: removing container" podUID="2e3a11a5-ece0-473f-9191-0e25881b179f" containerName="revoke-keys"
Nov 03 06:22:44.424575 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.424526    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e3a11a5-ece0-473f-9191-0e25881b179f" containerName="revoke-keys"
Nov 03 06:22:44.424676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.424663    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e3a11a5-ece0-473f-9191-0e25881b179f" containerName="revoke-keys"
Nov 03 06:22:44.428573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.428554    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.444588 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.444549    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"]
Nov 03 06:22:44.451548 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.451512    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:22:44.452450 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.452423    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:22:44.452604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.452515    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:22:44.455098 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.454261    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:22:44.509195 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.509148    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.509364 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.509285    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.509419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.509389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxvtv\" (UniqueName: \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.610251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.610211    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxvtv\" (UniqueName: \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.610461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.610278    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.610461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.610322    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.611141 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.611115    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.612951 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.612924    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.635510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.635475    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxvtv\" (UniqueName: \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\") pod \"maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:44.743857 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:44.743765    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:45.069196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:45.069086    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"]
Nov 03 06:22:45.071732 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:22:45.071700    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cdf8d8e_ccf9_4c7e_a5be_dc4ae8aa4cef.slice/crio-2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7 WatchSource:0}: Error finding container 2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7: Status 404 returned error can't find the container with id 2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7
Nov 03 06:22:45.224583 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:45.224543    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp" event={"ID":"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef","Type":"ContainerStarted","Data":"2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7"}
Nov 03 06:22:46.232897 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:46.232847    4528 generic.go:356] "Generic (PLEG): container finished" podID="3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" containerID="f01e6c25180bc9adab08d44a682ed97fa326203f845643cd34a2d5da0f6ee012" exitCode=0
Nov 03 06:22:46.233361 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:46.232975    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp" event={"ID":"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef","Type":"ContainerDied","Data":"f01e6c25180bc9adab08d44a682ed97fa326203f845643cd34a2d5da0f6ee012"}
Nov 03 06:22:47.389247 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.389216    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:47.445004 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.444960    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\") pod \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") "
Nov 03 06:22:47.445207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.445060    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\" (UniqueName: \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\") pod \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") "
Nov 03 06:22:47.445207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.445182    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\") pod \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\" (UID: \"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef\") "
Nov 03 06:22:47.445398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.445376    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca" pod "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" (UID: "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:22:47.445634 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.445613    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:47.447307 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.447259    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca" pod "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" (UID: "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:22:47.447426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.447391    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv" pod "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" (UID: "3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef"). InnerVolumeSpecName "kube-api-access-mxvtv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:22:47.547032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.546942    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:47.547032 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:47.546981    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxvtv\" (UniqueName: \"kubernetes.io/projected/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef-kube-api-access-mxvtv\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:22:48.247882 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:48.247850    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"
Nov 03 06:22:48.248084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:48.247889    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp" event={"ID":"3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef","Type":"ContainerDied","Data":"2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7"}
Nov 03 06:22:48.248084 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:48.247947    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a828a35d78cb82af0a8fd030083b6fe001d2d857ea9b38d6deb4a3bbb5b0cf7"
Nov 03 06:22:49.136001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:49.135964    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:22:54.890448 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:54.890369    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:22:54.921977 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:54.921938    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:22:56.974177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:56.974138    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"]
Nov 03 06:22:56.988589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:56.988551    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-7c2bc6-n2nxp"]
Nov 03 06:22:57.748542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:57.748509    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" path="/var/lib/kubelet/pods/3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef/volumes"
Nov 03 06:22:59.891494 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:59.891449    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" podUID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" containerName="payload-processing" containerID="cri-o://7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a" gracePeriod=25
Nov 03 06:22:59.923208 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:22:59.923149    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" podUID="340d1b46-b755-442e-b348-d4c9dd86a0c9" containerName="payload-pre-processing" containerID="cri-o://54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394" gracePeriod=25
Nov 03 06:23:00.179996 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.179967    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:23:00.211937 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.211908    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:23:00.286168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.286127    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\" (UniqueName: \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\") pod \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") "
Nov 03 06:23:00.286334 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.286303    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\") pod \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\" (UID: \"8c390c2a-0933-4d05-ae93-0a1d0f005c45\") "
Nov 03 06:23:00.286770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.286738    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume" pod "8c390c2a-0933-4d05-ae93-0a1d0f005c45" (UID: "8c390c2a-0933-4d05-ae93-0a1d0f005c45"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:23:00.288444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.288415    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6" pod "8c390c2a-0933-4d05-ae93-0a1d0f005c45" (UID: "8c390c2a-0933-4d05-ae93-0a1d0f005c45"). InnerVolumeSpecName "kube-api-access-nqgc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:23:00.326960 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.326918    4528 generic.go:356] "Generic (PLEG): container finished" podID="340d1b46-b755-442e-b348-d4c9dd86a0c9" containerID="54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394" exitCode=0
Nov 03 06:23:00.327114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.326971    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"
Nov 03 06:23:00.327114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.326999    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" event={"ID":"340d1b46-b755-442e-b348-d4c9dd86a0c9","Type":"ContainerDied","Data":"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"}
Nov 03 06:23:00.327114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.327059    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g" event={"ID":"340d1b46-b755-442e-b348-d4c9dd86a0c9","Type":"ContainerDied","Data":"393dcb154edc06800b9d213793faf6075b21d6ee89dcdf2c2321d410831f9d5e"}
Nov 03 06:23:00.327114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.327083    4528 scope.go:122] "RemoveContainer" containerID="54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"
Nov 03 06:23:00.328692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.328665    4528 generic.go:356] "Generic (PLEG): container finished" podID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" containerID="7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a" exitCode=0
Nov 03 06:23:00.328814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.328703    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" event={"ID":"8c390c2a-0933-4d05-ae93-0a1d0f005c45","Type":"ContainerDied","Data":"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"}
Nov 03 06:23:00.328814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.328710    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"
Nov 03 06:23:00.328814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.328744    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js" event={"ID":"8c390c2a-0933-4d05-ae93-0a1d0f005c45","Type":"ContainerDied","Data":"db8e7d789ec733b939a615bdcc1e231afaf45c8e41857d418635a7c498f1bcd5"}
Nov 03 06:23:00.337861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.337844    4528 scope.go:122] "RemoveContainer" containerID="54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"
Nov 03 06:23:00.338174 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:23:00.338157    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394\": container with ID starting with 54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394 not found: ID does not exist" containerID="54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"
Nov 03 06:23:00.338254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.338185    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394"} err="failed to get container status \"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394\": rpc error: code = NotFound desc = could not find container \"54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394\": container with ID starting with 54441f98d9ac583a299b8da6176456cbba69b60a4655fef6c568b268cb5e8394 not found: ID does not exist"
Nov 03 06:23:00.338254 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.338208    4528 scope.go:122] "RemoveContainer" containerID="7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"
Nov 03 06:23:00.348803 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.348773    4528 scope.go:122] "RemoveContainer" containerID="7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"
Nov 03 06:23:00.349153 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:23:00.349123    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a\": container with ID starting with 7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a not found: ID does not exist" containerID="7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"
Nov 03 06:23:00.349250 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.349161    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a"} err="failed to get container status \"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a\": rpc error: code = NotFound desc = could not find container \"7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a\": container with ID starting with 7d256425c389c933ee12dac47c3bf42da55b73ae49e99d9fdefff1609f6fd39a not found: ID does not exist"
Nov 03 06:23:00.384539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.384500    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:23:00.387489 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.387457    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\" (UniqueName: \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\") pod \"340d1b46-b755-442e-b348-d4c9dd86a0c9\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") "
Nov 03 06:23:00.387617 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.387509    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\") pod \"340d1b46-b755-442e-b348-d4c9dd86a0c9\" (UID: \"340d1b46-b755-442e-b348-d4c9dd86a0c9\") "
Nov 03 06:23:00.387898 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.387864    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume" pod "340d1b46-b755-442e-b348-d4c9dd86a0c9" (UID: "340d1b46-b755-442e-b348-d4c9dd86a0c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:23:00.388024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.387983    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgc6\" (UniqueName: \"kubernetes.io/projected/8c390c2a-0933-4d05-ae93-0a1d0f005c45-kube-api-access-nqgc6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:00.388024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.388002    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c390c2a-0933-4d05-ae93-0a1d0f005c45-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:00.388024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.388015    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/340d1b46-b755-442e-b348-d4c9dd86a0c9-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:00.388618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.388598    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-7c2bc6-86c5fbf86d22js"]
Nov 03 06:23:00.389698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.389673    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng" pod "340d1b46-b755-442e-b348-d4c9dd86a0c9" (UID: "340d1b46-b755-442e-b348-d4c9dd86a0c9"). InnerVolumeSpecName "kube-api-access-df4ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:23:00.489402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.489317    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df4ng\" (UniqueName: \"kubernetes.io/projected/340d1b46-b755-442e-b348-d4c9dd86a0c9-kube-api-access-df4ng\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:00.671157 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.671113    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:23:00.675585 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:00.675551    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-7c2bc6-7845fdcj8g"]
Nov 03 06:23:01.748732 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:01.748694    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="340d1b46-b755-442e-b348-d4c9dd86a0c9" path="/var/lib/kubelet/pods/340d1b46-b755-442e-b348-d4c9dd86a0c9/volumes"
Nov 03 06:23:01.749201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:01.749174    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" path="/var/lib/kubelet/pods/8c390c2a-0933-4d05-ae93-0a1d0f005c45/volumes"
Nov 03 06:23:04.156209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.156167    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" podUID="9971edd1-8208-4ea2-a118-e33ca6109574" containerName="main" containerID="cri-o://216abf44105e93a825e1f701970b3bae21492476875719fa2e320157dce3fda7" gracePeriod=45
Nov 03 06:23:04.361391 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.361355    4528 generic.go:356] "Generic (PLEG): container finished" podID="9971edd1-8208-4ea2-a118-e33ca6109574" containerID="216abf44105e93a825e1f701970b3bae21492476875719fa2e320157dce3fda7" exitCode=0
Nov 03 06:23:04.361611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.361406    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" event={"ID":"9971edd1-8208-4ea2-a118-e33ca6109574","Type":"ContainerDied","Data":"216abf44105e93a825e1f701970b3bae21492476875719fa2e320157dce3fda7"}
Nov 03 06:23:04.421534 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.421454    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:23:04.534042 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.533995    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.534253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534082    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.534253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534152    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.534253 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534192    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\" (UniqueName: \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.534485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534449    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:23:04.534610 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534586    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:23:04.534681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.534602    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:23:04.535028 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.535005    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\" (UniqueName: \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.535144 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.535127    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\") pod \"9971edd1-8208-4ea2-a118-e33ca6109574\" (UID: \"9971edd1-8208-4ea2-a118-e33ca6109574\") "
Nov 03 06:23:04.535573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.535550    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:04.535698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.535575    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:04.535698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.535589    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:04.537365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.537330    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "kube-api-access-z8c9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:23:04.537478 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.537424    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:23:04.537870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.537838    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm" pod "9971edd1-8208-4ea2-a118-e33ca6109574" (UID: "9971edd1-8208-4ea2-a118-e33ca6109574"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:23:04.636591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.636548    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8c9w\" (UniqueName: \"kubernetes.io/projected/9971edd1-8208-4ea2-a118-e33ca6109574-kube-api-access-z8c9w\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:04.636591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.636586    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9971edd1-8208-4ea2-a118-e33ca6109574-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:04.636591 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:04.636596    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9971edd1-8208-4ea2-a118-e33ca6109574-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:05.370580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.370544    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"
Nov 03 06:23:05.371012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.370547    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm" event={"ID":"9971edd1-8208-4ea2-a118-e33ca6109574","Type":"ContainerDied","Data":"3b6db79c92616fca8e3c3233f1896ef93690f079fcf100bcc6c3c18aa83178b1"}
Nov 03 06:23:05.371012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.370684    4528 scope.go:122] "RemoveContainer" containerID="216abf44105e93a825e1f701970b3bae21492476875719fa2e320157dce3fda7"
Nov 03 06:23:05.421755 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.421721    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:23:05.432775 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.432739    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-7c2bc6/simulator-single-main-7c2bc6-kserve-7b7d9979f7-nc9pm"]
Nov 03 06:23:05.747718 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:05.747636    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9971edd1-8208-4ea2-a118-e33ca6109574" path="/var/lib/kubelet/pods/9971edd1-8208-4ea2-a118-e33ca6109574/volumes"
Nov 03 06:23:08.495458 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495420    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:23:08.495879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495866    4528 cpu_manager.go:414] "containerMap: removing container" podUID="3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" containerName="revoke-keys"
Nov 03 06:23:08.495879 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495877    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" containerName="revoke-keys"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495898    4528 cpu_manager.go:414] "containerMap: removing container" podUID="340d1b46-b755-442e-b348-d4c9dd86a0c9" containerName="payload-pre-processing"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495904    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="340d1b46-b755-442e-b348-d4c9dd86a0c9" containerName="payload-pre-processing"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495924    4528 cpu_manager.go:414] "containerMap: removing container" podUID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" containerName="payload-processing"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495929    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" containerName="payload-processing"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495946    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9971edd1-8208-4ea2-a118-e33ca6109574" containerName="main"
Nov 03 06:23:08.495966 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.495952    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9971edd1-8208-4ea2-a118-e33ca6109574" containerName="main"
Nov 03 06:23:08.496211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.496010    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="340d1b46-b755-442e-b348-d4c9dd86a0c9" containerName="payload-pre-processing"
Nov 03 06:23:08.496211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.496022    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c390c2a-0933-4d05-ae93-0a1d0f005c45" containerName="payload-processing"
Nov 03 06:23:08.496211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.496031    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9971edd1-8208-4ea2-a118-e33ca6109574" containerName="main"
Nov 03 06:23:08.496211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.496036    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="3cdf8d8e-ccf9-4c7e-a5be-dc4ae8aa4cef" containerName="revoke-keys"
Nov 03 06:23:08.500828 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.500807    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.518127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.518100    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:23:08.526041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.526007    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-6d432a37-kserve-self-signed-certs\""
Nov 03 06:23:08.573980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.573947    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.573980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.573982    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.574202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.574002    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.574202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.574030    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.574202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.574079    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.574202 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.574183    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw9s5\" (UniqueName: \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675252    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675305    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675353    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw9s5\" (UniqueName: \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675505    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675534    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.675648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675562    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.676012 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.675982    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.676086 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.676061    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.676278 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.676256    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.678669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.678641    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.681468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.681410    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.698847 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.698780    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw9s5\" (UniqueName: \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\") pod \"e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") " pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:08.814444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:08.814410    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:09.119664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:09.119638    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:23:09.121032 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:23:09.121003    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9feea35f_ba47_489a_b5b9_d76a85fc6f59.slice/crio-13abc4f136c9cc340e0829a0812e8e415d11e30635ba5fa3162fce1ef584ab85 WatchSource:0}: Error finding container 13abc4f136c9cc340e0829a0812e8e415d11e30635ba5fa3162fce1ef584ab85: Status 404 returned error can't find the container with id 13abc4f136c9cc340e0829a0812e8e415d11e30635ba5fa3162fce1ef584ab85
Nov 03 06:23:09.398491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:09.398396    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" event={"ID":"9feea35f-ba47-489a-b5b9-d76a85fc6f59","Type":"ContainerStarted","Data":"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"}
Nov 03 06:23:09.398491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:09.398447    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" event={"ID":"9feea35f-ba47-489a-b5b9-d76a85fc6f59","Type":"ContainerStarted","Data":"13abc4f136c9cc340e0829a0812e8e415d11e30635ba5fa3162fce1ef584ab85"}
Nov 03 06:23:09.439120 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:09.439059    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" podStartSLOduration=1.439041938 podStartE2EDuration="1.439041938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:23:09.438100231 +0000 UTC m=+1677.849497310" watchObservedRunningTime="2026-11-03 06:23:09.439041938 +0000 UTC m=+1677.850439033"
Nov 03 06:23:09.815163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:09.815126    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:13.668819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:13.665922    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:23:13.668819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:13.666016    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:23:13.823818 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:23:13.822008    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:23:15.382714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:15.382672    4528 scope.go:122] "RemoveContainer" containerID="0e9b29ea3722898479d0c78b57f00da48703e9c2789dd15249c09d56ad4ba237"
Nov 03 06:23:15.396521 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:15.396499    4528 scope.go:122] "RemoveContainer" containerID="f51b5d3dc0fe1f22b432828dfb3fef691bfa43fb81fd67a3d858d042bb8bac03"
Nov 03 06:23:15.406869 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:15.406849    4528 scope.go:122] "RemoveContainer" containerID="257577aabbf9a42dbca8e51664d7112cda248d939821feea81ab0337b901ae4f"
Nov 03 06:23:18.815444 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:18.815408    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:18.828129 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:18.828099    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:19.516435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:19.516406    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:23:43.693780 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:23:43.693741    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:23:46.033766 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:46.033730    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"]
Nov 03 06:23:46.044920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:46.044867    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dfc7248-xljgn"]
Nov 03 06:23:47.748123 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:47.748083    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="907d069a-e8b4-413f-bc7a-bf4b937730c3" path="/var/lib/kubelet/pods/907d069a-e8b4-413f-bc7a-bf4b937730c3/volumes"
Nov 03 06:23:49.604312 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.604263    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"]
Nov 03 06:23:49.608811 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.608772    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.632059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.632027    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"]
Nov 03 06:23:49.636460 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.636419    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:23:49.637163 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.637115    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:23:49.637704 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.637677    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:23:49.638170 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.637975    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:23:49.645573 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.645543    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.645709 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.645599    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tghq6\" (UniqueName: \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.645850 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.645825    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.747074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.747021    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.748211 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.748187    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.748416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.748397    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tghq6\" (UniqueName: \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.748713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.748098    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.753903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.753681    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.783594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.783557    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tghq6\" (UniqueName: \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\") pod \"maas-api-revoke-sub-452b6d7b-qjlx2\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:49.940365 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:49.940254    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:50.313305 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:50.313209    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"]
Nov 03 06:23:50.790477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:50.790388    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2" event={"ID":"807febcc-ec5a-477c-a56f-f45956e55c40","Type":"ContainerStarted","Data":"edd45ad338b71a35a6a401a54f4c604eca1b3daf4919167fe4a0daa6e58d1e55"}
Nov 03 06:23:51.898875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:51.897885    4528 generic.go:356] "Generic (PLEG): container finished" podID="807febcc-ec5a-477c-a56f-f45956e55c40" containerID="e9253fe45a5b07d22ecc7f2df94238e59d5c21dd9ab938814de05c47d8f09d78" exitCode=0
Nov 03 06:23:51.898875 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:51.898044    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2" event={"ID":"807febcc-ec5a-477c-a56f-f45956e55c40","Type":"ContainerDied","Data":"e9253fe45a5b07d22ecc7f2df94238e59d5c21dd9ab938814de05c47d8f09d78"}
Nov 03 06:23:52.705814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:52.705282    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-zw8r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:23:52.705814 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:52.705347    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-zw8r9" podUID="bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.93:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:23:53.041818 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.041775    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:53.171580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.171548    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\" (UniqueName: \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\") pod \"807febcc-ec5a-477c-a56f-f45956e55c40\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") "
Nov 03 06:23:53.171751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.171594    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\") pod \"807febcc-ec5a-477c-a56f-f45956e55c40\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") "
Nov 03 06:23:53.171751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.171669    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\") pod \"807febcc-ec5a-477c-a56f-f45956e55c40\" (UID: \"807febcc-ec5a-477c-a56f-f45956e55c40\") "
Nov 03 06:23:53.172143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.172119    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca" pod "807febcc-ec5a-477c-a56f-f45956e55c40" (UID: "807febcc-ec5a-477c-a56f-f45956e55c40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:23:53.173823 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.173772    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca" pod "807febcc-ec5a-477c-a56f-f45956e55c40" (UID: "807febcc-ec5a-477c-a56f-f45956e55c40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:23:53.173924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.173857    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6" pod "807febcc-ec5a-477c-a56f-f45956e55c40" (UID: "807febcc-ec5a-477c-a56f-f45956e55c40"). InnerVolumeSpecName "kube-api-access-tghq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:23:53.272962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.272877    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tghq6\" (UniqueName: \"kubernetes.io/projected/807febcc-ec5a-477c-a56f-f45956e55c40-kube-api-access-tghq6\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:53.272962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.272909    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:53.272962 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.272919    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/807febcc-ec5a-477c-a56f-f45956e55c40-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:23:53.911265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.911231    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"
Nov 03 06:23:53.911415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.911263    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2" event={"ID":"807febcc-ec5a-477c-a56f-f45956e55c40","Type":"ContainerDied","Data":"edd45ad338b71a35a6a401a54f4c604eca1b3daf4919167fe4a0daa6e58d1e55"}
Nov 03 06:23:53.911415 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:23:53.911329    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edd45ad338b71a35a6a401a54f4c604eca1b3daf4919167fe4a0daa6e58d1e55"
Nov 03 06:24:00.836925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:00.836891    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:24:06.706787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.706752    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"]
Nov 03 06:24:06.707328 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.707311    4528 cpu_manager.go:414] "containerMap: removing container" podUID="807febcc-ec5a-477c-a56f-f45956e55c40" containerName="revoke-keys"
Nov 03 06:24:06.707377 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.707329    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="807febcc-ec5a-477c-a56f-f45956e55c40" containerName="revoke-keys"
Nov 03 06:24:06.707433 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.707423    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="807febcc-ec5a-477c-a56f-f45956e55c40" containerName="revoke-keys"
Nov 03 06:24:06.710604 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.710583    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.734542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.734505    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"]
Nov 03 06:24:06.738164 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.738136    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:24:06.741891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.740145    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:24:06.741891 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.740189    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:24:06.751155 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.751133    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:24:06.790516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.790476    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.790716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.790625    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.790716 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.790661    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p59jv\" (UniqueName: \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.892043 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.891999    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.892231 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.892064    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.892231 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.892106    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p59jv\" (UniqueName: \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.893011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.892857    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.894758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.894730    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:06.916047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:06.916012    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p59jv\" (UniqueName: \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\") pod \"maas-api-revoke-sub-3e31a506-fn8hg\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:07.026502 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:07.026412    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:07.328989 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:07.328964    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"]
Nov 03 06:24:07.331608 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:24:07.331573    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96928d09_a89d_444c_a690_70aa5155735f.slice/crio-6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5 WatchSource:0}: Error finding container 6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5: Status 404 returned error can't find the container with id 6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5
Nov 03 06:24:08.064298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:08.064250    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg" event={"ID":"96928d09-a89d-444c-a690-70aa5155735f","Type":"ContainerStarted","Data":"65d86802be3a5da9413a3035f9418b155ac8492572d33f04b60ec1d7d50fcbab"}
Nov 03 06:24:08.064870 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:08.064328    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg" event={"ID":"96928d09-a89d-444c-a690-70aa5155735f","Type":"ContainerStarted","Data":"6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5"}
Nov 03 06:24:08.104785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:08.104719    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg" podStartSLOduration=2.104701286 podStartE2EDuration="2.104701286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:08.10387903 +0000 UTC m=+1736.515276109" watchObservedRunningTime="2026-11-03 06:24:08.104701286 +0000 UTC m=+1736.516098365"
Nov 03 06:24:09.092105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:09.092066    4528 generic.go:356] "Generic (PLEG): container finished" podID="96928d09-a89d-444c-a690-70aa5155735f" containerID="65d86802be3a5da9413a3035f9418b155ac8492572d33f04b60ec1d7d50fcbab" exitCode=0
Nov 03 06:24:09.096868 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:09.096826    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg" event={"ID":"96928d09-a89d-444c-a690-70aa5155735f","Type":"ContainerDied","Data":"65d86802be3a5da9413a3035f9418b155ac8492572d33f04b60ec1d7d50fcbab"}
Nov 03 06:24:10.586568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.586534    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:10.726186 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.726148    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\") pod \"96928d09-a89d-444c-a690-70aa5155735f\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") "
Nov 03 06:24:10.726381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.726229    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\" (UniqueName: \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\") pod \"96928d09-a89d-444c-a690-70aa5155735f\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") "
Nov 03 06:24:10.726381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.726366    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\") pod \"96928d09-a89d-444c-a690-70aa5155735f\" (UID: \"96928d09-a89d-444c-a690-70aa5155735f\") "
Nov 03 06:24:10.726741 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.726708    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca" pod "96928d09-a89d-444c-a690-70aa5155735f" (UID: "96928d09-a89d-444c-a690-70aa5155735f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:24:10.728406 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.728383    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv" pod "96928d09-a89d-444c-a690-70aa5155735f" (UID: "96928d09-a89d-444c-a690-70aa5155735f"). InnerVolumeSpecName "kube-api-access-p59jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:24:10.728576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.728556    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca" pod "96928d09-a89d-444c-a690-70aa5155735f" (UID: "96928d09-a89d-444c-a690-70aa5155735f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:24:10.827570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.827485    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96928d09-a89d-444c-a690-70aa5155735f-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:10.827570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.827516    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96928d09-a89d-444c-a690-70aa5155735f-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:10.827570 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:10.827529    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p59jv\" (UniqueName: \"kubernetes.io/projected/96928d09-a89d-444c-a690-70aa5155735f-kube-api-access-p59jv\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:11.139426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:11.139337    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"
Nov 03 06:24:11.139426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:11.139376    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg" event={"ID":"96928d09-a89d-444c-a690-70aa5155735f","Type":"ContainerDied","Data":"6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5"}
Nov 03 06:24:11.139426 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:11.139424    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f84aeee81ed8d8b52c9dadaa6d5d74532142340383d66c80a0a62990e5731e5"
Nov 03 06:24:13.693081 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:24:13.693042    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:24:15.454353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:15.454319    4528 scope.go:122] "RemoveContainer" containerID="1974c4b52a0fd771cb9e255dd4f1820bfec9d08dd95603dbfe532b4bca0c899a"
Nov 03 06:24:15.857660 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:15.857596    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" podUID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" containerName="main" containerID="cri-o://2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e" gracePeriod=45
Nov 03 06:24:16.117514 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.117451    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:24:16.170853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.170771    4528 generic.go:356] "Generic (PLEG): container finished" podID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" containerID="2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e" exitCode=0
Nov 03 06:24:16.170853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.170834    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" event={"ID":"9feea35f-ba47-489a-b5b9-d76a85fc6f59","Type":"ContainerDied","Data":"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"}
Nov 03 06:24:16.171057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.170883    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n" event={"ID":"9feea35f-ba47-489a-b5b9-d76a85fc6f59","Type":"ContainerDied","Data":"13abc4f136c9cc340e0829a0812e8e415d11e30635ba5fa3162fce1ef584ab85"}
Nov 03 06:24:16.171057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.170904    4528 scope.go:122] "RemoveContainer" containerID="2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"
Nov 03 06:24:16.171057 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.170847    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"
Nov 03 06:24:16.182685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.182659    4528 scope.go:122] "RemoveContainer" containerID="2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"
Nov 03 06:24:16.183127 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:24:16.183103    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e\": container with ID starting with 2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e not found: ID does not exist" containerID="2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"
Nov 03 06:24:16.183230 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.183144    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e"} err="failed to get container status \"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e\": rpc error: code = NotFound desc = could not find container \"2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e\": container with ID starting with 2573f9f86e47b1fc3bd93f3343e0a9154633c0a208d751c9d2a621b49f8a416e not found: ID does not exist"
Nov 03 06:24:16.285235 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285200    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\" (UniqueName: \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285257    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285348    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\" (UniqueName: \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285453 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285407    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285527    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285564    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\") pod \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\" (UID: \"9feea35f-ba47-489a-b5b9-d76a85fc6f59\") "
Nov 03 06:24:16.285637 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285614    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:24:16.285831 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:24:16.285930 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285909    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:24:16.285979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285942    4528 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tmp-dir\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.285979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.285963    4528 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-model-cache\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.287681 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.287654    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:24:16.287782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.287718    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "kube-api-access-sw9s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:24:16.287782 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.287730    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs" pod "9feea35f-ba47-489a-b5b9-d76a85fc6f59" (UID: "9feea35f-ba47-489a-b5b9-d76a85fc6f59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:24:16.386763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.386662    4528 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-home\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.386763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.386699    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw9s5\" (UniqueName: \"kubernetes.io/projected/9feea35f-ba47-489a-b5b9-d76a85fc6f59-kube-api-access-sw9s5\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.386763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.386714    4528 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9feea35f-ba47-489a-b5b9-d76a85fc6f59-tls-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.386763 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.386726    4528 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9feea35f-ba47-489a-b5b9-d76a85fc6f59-dshm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:16.515724 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.515685    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:24:16.527127 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:16.527093    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-6d432a37-kserve-7657567d77-lr69n"]
Nov 03 06:24:17.747476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:17.747443    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" path="/var/lib/kubelet/pods/9feea35f-ba47-489a-b5b9-d76a85fc6f59/volumes"
Nov 03 06:24:20.041836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:20.041757    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"]
Nov 03 06:24:20.053257 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:20.053216    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zvmlx"]
Nov 03 06:24:21.748283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:21.748235    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75d78c5c-550b-46d2-8916-14d4c6828ffb" path="/var/lib/kubelet/pods/75d78c5c-550b-46d2-8916-14d4c6828ffb/volumes"
Nov 03 06:24:30.272924 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.272889    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"]
Nov 03 06:24:30.273354 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273341    4528 cpu_manager.go:414] "containerMap: removing container" podUID="96928d09-a89d-444c-a690-70aa5155735f" containerName="revoke-keys"
Nov 03 06:24:30.273399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273355    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96928d09-a89d-444c-a690-70aa5155735f" containerName="revoke-keys"
Nov 03 06:24:30.273399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273373    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" containerName="main"
Nov 03 06:24:30.273399 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273380    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" containerName="main"
Nov 03 06:24:30.273496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273461    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="96928d09-a89d-444c-a690-70aa5155735f" containerName="revoke-keys"
Nov 03 06:24:30.273496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.273474    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9feea35f-ba47-489a-b5b9-d76a85fc6f59" containerName="main"
Nov 03 06:24:30.276609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.276587    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.292843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.292772    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:24:30.304214 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.304181    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:24:30.304445 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.304281    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:24:30.304655 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.304620    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:24:30.308769 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.308739    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"]
Nov 03 06:24:30.328602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.328557    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.328786 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.328673    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prwd7\" (UniqueName: \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.328861 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.328838    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.430048 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.430006    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.430265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.430115    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.430265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.430159    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prwd7\" (UniqueName: \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.430871 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.430842    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.432479 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.432456    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.453135 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.453102    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prwd7\" (UniqueName: \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\") pod \"maas-api-revoke-sub-af191ac6-mvhpp\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.591065 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.591027    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:30.892979 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:30.892949    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"]
Nov 03 06:24:30.894169 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:24:30.894134    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41dec697_134c_40b5_99b5_77846f9975f5.slice/crio-aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3 WatchSource:0}: Error finding container aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3: Status 404 returned error can't find the container with id aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3
Nov 03 06:24:31.473488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:31.473390    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp" event={"ID":"41dec697-134c-40b5-99b5-77846f9975f5","Type":"ContainerStarted","Data":"fc583bcdb126e9965887a2dafa65e828d785af5e799770a5f8f6a09cd863c3b4"}
Nov 03 06:24:31.473488 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:31.473449    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp" event={"ID":"41dec697-134c-40b5-99b5-77846f9975f5","Type":"ContainerStarted","Data":"aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3"}
Nov 03 06:24:31.513736 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:31.513678    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp" podStartSLOduration=1.513665194 podStartE2EDuration="1.513665194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:31.512381908 +0000 UTC m=+1759.923778989" watchObservedRunningTime="2026-11-03 06:24:31.513665194 +0000 UTC m=+1759.925062272"
Nov 03 06:24:32.480641 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:32.480599    4528 generic.go:356] "Generic (PLEG): container finished" podID="41dec697-134c-40b5-99b5-77846f9975f5" containerID="fc583bcdb126e9965887a2dafa65e828d785af5e799770a5f8f6a09cd863c3b4" exitCode=0
Nov 03 06:24:32.481082 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:32.480686    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp" event={"ID":"41dec697-134c-40b5-99b5-77846f9975f5","Type":"ContainerDied","Data":"fc583bcdb126e9965887a2dafa65e828d785af5e799770a5f8f6a09cd863c3b4"}
Nov 03 06:24:33.624691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.624665    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:33.764049 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.763979    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\" (UniqueName: \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\") pod \"41dec697-134c-40b5-99b5-77846f9975f5\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") "
Nov 03 06:24:33.764200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.764076    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\") pod \"41dec697-134c-40b5-99b5-77846f9975f5\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") "
Nov 03 06:24:33.764200 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.764161    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\") pod \"41dec697-134c-40b5-99b5-77846f9975f5\" (UID: \"41dec697-134c-40b5-99b5-77846f9975f5\") "
Nov 03 06:24:33.764555 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.764509    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca" pod "41dec697-134c-40b5-99b5-77846f9975f5" (UID: "41dec697-134c-40b5-99b5-77846f9975f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:24:33.766142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.766110    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7" pod "41dec697-134c-40b5-99b5-77846f9975f5" (UID: "41dec697-134c-40b5-99b5-77846f9975f5"). InnerVolumeSpecName "kube-api-access-prwd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:24:33.766239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.766206    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca" pod "41dec697-134c-40b5-99b5-77846f9975f5" (UID: "41dec697-134c-40b5-99b5-77846f9975f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:24:33.866040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.866004    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prwd7\" (UniqueName: \"kubernetes.io/projected/41dec697-134c-40b5-99b5-77846f9975f5-kube-api-access-prwd7\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:33.866040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.866034    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:33.866040 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:33.866047    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41dec697-134c-40b5-99b5-77846f9975f5-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:24:34.494650 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:34.494606    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp" event={"ID":"41dec697-134c-40b5-99b5-77846f9975f5","Type":"ContainerDied","Data":"aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3"}
Nov 03 06:24:34.494890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:34.494664    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadf7f3a01621d3db3b0efbbbc891a42de4199aadbdc47dcdd5c42853d94eed3"
Nov 03 06:24:34.494890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:34.494621    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"
Nov 03 06:24:43.691288 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:24:43.691244    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:24:49.879000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.878967    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:24:49.879499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.879446    4528 cpu_manager.go:414] "containerMap: removing container" podUID="41dec697-134c-40b5-99b5-77846f9975f5" containerName="revoke-keys"
Nov 03 06:24:49.879499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.879460    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41dec697-134c-40b5-99b5-77846f9975f5" containerName="revoke-keys"
Nov 03 06:24:49.879589 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.879559    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="41dec697-134c-40b5-99b5-77846f9975f5" containerName="revoke-keys"
Nov 03 06:24:49.882999 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.882979    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.901265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.901243    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-e41e4c4d-openshift-default-dockercfg-4l9d9\""
Nov 03 06:24:49.917366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.917340    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:24:49.932176 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932151    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932217    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932330    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932374    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp46j\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932463    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932525    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932545    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932656 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932612    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:49.932780 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:49.932678    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.034414 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.034173    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.034699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.034677    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.034876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.034858    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035190    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035317 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035304    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp46j\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035399    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035539 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035524    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035648 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035634    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.035757 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.035737    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.036166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.036143    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.036551 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.036532    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.037427 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.037388    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.037939 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.037918    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.041418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.041392    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.048179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.048103    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.059119 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.059084    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp46j\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.060344 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.060318    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\") pod \"e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") " pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.196906 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.196825    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:50.517337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.517313    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:24:50.518353 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:24:50.518328    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c70eb8_b2a7_4758_af1b_fec537d9c18e.slice/crio-0bec62ac0120e04767dfd00d0a541f50ed3ee72656a75a56b207d81f70372cf2 WatchSource:0}: Error finding container 0bec62ac0120e04767dfd00d0a541f50ed3ee72656a75a56b207d81f70372cf2: Status 404 returned error can't find the container with id 0bec62ac0120e04767dfd00d0a541f50ed3ee72656a75a56b207d81f70372cf2
Nov 03 06:24:50.520484 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.520449    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:24:50.520571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.520519    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:24:50.520571 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.520555    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:24:50.599626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.599595    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerStarted","Data":"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"}
Nov 03 06:24:50.599741 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.599662    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerStarted","Data":"0bec62ac0120e04767dfd00d0a541f50ed3ee72656a75a56b207d81f70372cf2"}
Nov 03 06:24:50.647544 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:50.647492    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podStartSLOduration=1.647473083 podStartE2EDuration="1.647473083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:50.643759629 +0000 UTC m=+1779.055156708" watchObservedRunningTime="2026-11-03 06:24:50.647473083 +0000 UTC m=+1779.058870159"
Nov 03 06:24:51.197300 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:51.197267    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:51.202390 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:51.202363    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:51.606315 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:51.606278    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:51.607152 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:51.607130    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:24:54.295561 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.295518    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:24:54.299470 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.299447    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.319310 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.319283    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:24:54.326398 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.326374    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e41e4c4d\""
Nov 03 06:24:54.326536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.326415    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e41e4c4d-dockercfg-vmqb5\""
Nov 03 06:24:54.381623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.381573    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.381880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.381767    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgdts\" (UniqueName: \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.392360 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.392328    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:24:54.397095 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.397067    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.419611 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.419575    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:24:54.482852 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.482811    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.483034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.482873    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.483034 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.482996    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5wnm\" (UniqueName: \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.483184 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.483037    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgdts\" (UniqueName: \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.483602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.483577    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.506380 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.506347    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgdts\" (UniqueName: \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\") pod \"payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.584108 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.584067    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.584325 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.584184    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5wnm\" (UniqueName: \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.584915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.584896    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.606969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.606941    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5wnm\" (UniqueName: \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\") pod \"payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") " pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.613833 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.613806    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:54.716331 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.716291    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:54.913094 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:54.913066    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:24:54.913853 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:24:54.913827    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a605277_e51d_44dd_922d_56642bd91bac.slice/crio-a5f6987c7434de42813651fd0f7b80129a121477870ce7e0c064c6859b89ad52 WatchSource:0}: Error finding container a5f6987c7434de42813651fd0f7b80129a121477870ce7e0c064c6859b89ad52: Status 404 returned error can't find the container with id a5f6987c7434de42813651fd0f7b80129a121477870ce7e0c064c6859b89ad52
Nov 03 06:24:55.028369 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.028337    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:24:55.029204 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:24:55.029175    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8d4f73b_d977_45bc_b4b2_d5eea6318274.slice/crio-84ee56b2aca11b37b5a5afa86a1e641535b241e74cacd5e987e43d6f95b215a2 WatchSource:0}: Error finding container 84ee56b2aca11b37b5a5afa86a1e641535b241e74cacd5e987e43d6f95b215a2: Status 404 returned error can't find the container with id 84ee56b2aca11b37b5a5afa86a1e641535b241e74cacd5e987e43d6f95b215a2
Nov 03 06:24:55.632940 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.632847    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" event={"ID":"c8d4f73b-d977-45bc-b4b2-d5eea6318274","Type":"ContainerStarted","Data":"a914f5ff0133c01441dab7685b0528fb5e09ec69225904764ac5b97f4c88fa8a"}
Nov 03 06:24:55.632940 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.632907    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" event={"ID":"c8d4f73b-d977-45bc-b4b2-d5eea6318274","Type":"ContainerStarted","Data":"84ee56b2aca11b37b5a5afa86a1e641535b241e74cacd5e987e43d6f95b215a2"}
Nov 03 06:24:55.633538 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.633212    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:24:55.634595 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.634572    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" event={"ID":"0a605277-e51d-44dd-922d-56642bd91bac","Type":"ContainerStarted","Data":"7a6cc73bf601dacd0b6c86594f6d91c3a45f9a6b8caee0074043335923ad96eb"}
Nov 03 06:24:55.634695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.634607    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" event={"ID":"0a605277-e51d-44dd-922d-56642bd91bac","Type":"ContainerStarted","Data":"a5f6987c7434de42813651fd0f7b80129a121477870ce7e0c064c6859b89ad52"}
Nov 03 06:24:55.634695 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.634636    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:24:55.673545 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.673493    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" podStartSLOduration=1.673479884 podStartE2EDuration="1.673479884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:55.671905295 +0000 UTC m=+1784.083302373" watchObservedRunningTime="2026-11-03 06:24:55.673479884 +0000 UTC m=+1784.084876961"
Nov 03 06:24:55.712329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:24:55.712254    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" podStartSLOduration=1.712229159 podStartE2EDuration="1.712229159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:24:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:24:55.710284721 +0000 UTC m=+1784.121681797" watchObservedRunningTime="2026-11-03 06:24:55.712229159 +0000 UTC m=+1784.123626236"
Nov 03 06:25:06.643092 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:06.643059    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:25:06.645520 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:06.643128    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:25:07.338054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.338016    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": read tcp 10.132.0.2:52304->10.132.0.189:15021: read: connection reset by peer" start-of-body=
Nov 03 06:25:07.338228 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.338130    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": read tcp 10.132.0.2:52304->10.132.0.189:15021: read: connection reset by peer"
Nov 03 06:25:07.748191 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.748108    4528 generic.go:356] "Generic (PLEG): container finished" podID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerID="33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a" exitCode=0
Nov 03 06:25:07.748564 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.748502    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerDied","Data":"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"}
Nov 03 06:25:07.748858 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.748843    4528 scope.go:122] "RemoveContainer" containerID="33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"
Nov 03 06:25:07.749972 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.749943    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:25:07.750035 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.750014    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:25:07.750080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:07.750045    4528 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236072Ki","pods":"250"}
Nov 03 06:25:08.267281 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.267242    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"]
Nov 03 06:25:08.270330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.270305    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.299066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.297011    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:25:08.299066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.297265    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:25:08.299066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.297345    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:25:08.299066 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.298091    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-rxh5n\""
Nov 03 06:25:08.305679 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.305629    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"]
Nov 03 06:25:08.370491 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.370456    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.370682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.370511    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.370682 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.370653    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.472101 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.472062    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.472298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.472121    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.472298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.472181    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.472904 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.472883    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.474618 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.474588    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:08.501197 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:08.501162    4528 projected.go:196] Error preparing data for projected volume kube-api-access-hz7dl for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 06:25:08.501422 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:08.501255    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl podName:a54825c9-7737-434d-86c3-11c68aa77e3e nodeName:}" failed. No retries permitted until 2026-11-03 06:25:09.001233994 +0000 UTC m=+1797.412631067 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hz7dl" (UniqueName: "kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl") pod "maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" (UID: "a54825c9-7737-434d-86c3-11c68aa77e3e") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 06:25:08.763734 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:08.763675    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerStarted","Data":"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"}
Nov 03 06:25:09.090843 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:09.090775    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:09.105646 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:09.105610    4528 projected.go:196] Error preparing data for projected volume kube-api-access-hz7dl for pod odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh: failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 06:25:09.105863 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:09.105684    4528 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl podName:a54825c9-7737-434d-86c3-11c68aa77e3e nodeName:}" failed. No retries permitted until 2026-11-03 06:25:10.105668165 +0000 UTC m=+1798.517065221 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hz7dl" (UniqueName: "kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl") pod "maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" (UID: "a54825c9-7737-434d-86c3-11c68aa77e3e") : failed to fetch token: serviceaccounts "maas-api-cleanup" not found
Nov 03 06:25:09.201001 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:09.200959    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:09.201350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:09.201325    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 06:25:09.201402 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:09.201378    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 06:25:10.110483 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.110442    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:10.133428 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.133396    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:10.201227 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.201185    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 06:25:10.201416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.201246    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 06:25:10.413766 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.413678    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" secret="" err="secret \"maas-api-cleanup-dockercfg-rxh5n\" not found"
Nov 03 06:25:10.413766 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.413743    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:10.724677 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.724642    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"]
Nov 03 06:25:10.725435 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:25:10.725401    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda54825c9_7737_434d_86c3_11c68aa77e3e.slice/crio-df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a WatchSource:0}: Error finding container df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a: Status 404 returned error can't find the container with id df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a
Nov 03 06:25:10.802627 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:10.802589    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" event={"ID":"a54825c9-7737-434d-86c3-11c68aa77e3e","Type":"ContainerStarted","Data":"df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a"}
Nov 03 06:25:11.200276 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:11.200228    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 06:25:11.201827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:11.200846    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 06:25:11.813474 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:11.813441    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" event={"ID":"a54825c9-7737-434d-86c3-11c68aa77e3e","Type":"ContainerStarted","Data":"35ce5ae22edb398c8757b008e6d06b3e51244750157ec397f250f99f0e5f84a1"}
Nov 03 06:25:11.813742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:11.813716    4528 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" secret="" err="secret \"maas-api-cleanup-dockercfg-rxh5n\" not found"
Nov 03 06:25:11.856209 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:11.856151    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" podStartSLOduration=3.856137964 podStartE2EDuration="3.856137964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:25:11.852691135 +0000 UTC m=+1800.264088212" watchObservedRunningTime="2026-11-03 06:25:11.856137964 +0000 UTC m=+1800.267535041"
Nov 03 06:25:12.201265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:12.201221    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 06:25:12.201768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:12.201289    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 06:25:12.823160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:12.823123    4528 generic.go:356] "Generic (PLEG): container finished" podID="a54825c9-7737-434d-86c3-11c68aa77e3e" containerID="35ce5ae22edb398c8757b008e6d06b3e51244750157ec397f250f99f0e5f84a1" exitCode=0
Nov 03 06:25:12.823345 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:12.823211    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" event={"ID":"a54825c9-7737-434d-86c3-11c68aa77e3e","Type":"ContainerDied","Data":"35ce5ae22edb398c8757b008e6d06b3e51244750157ec397f250f99f0e5f84a1"}
Nov 03 06:25:13.201417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:13.201313    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused" start-of-body=
Nov 03 06:25:13.201417 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:13.201388    4528 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.189:15021/healthz/ready\": dial tcp 10.132.0.189:15021: connect: connection refused"
Nov 03 06:25:13.692259 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:13.692212    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:25:13.975976 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:13.975941    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:14.053385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.053346    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\") pod \"a54825c9-7737-434d-86c3-11c68aa77e3e\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") "
Nov 03 06:25:14.053603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.053445    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\") pod \"a54825c9-7737-434d-86c3-11c68aa77e3e\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") "
Nov 03 06:25:14.053690 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.053601    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") pod \"a54825c9-7737-434d-86c3-11c68aa77e3e\" (UID: \"a54825c9-7737-434d-86c3-11c68aa77e3e\") "
Nov 03 06:25:14.053876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.053845    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca" pod "a54825c9-7737-434d-86c3-11c68aa77e3e" (UID: "a54825c9-7737-434d-86c3-11c68aa77e3e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:25:14.054003 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.053983    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:14.055568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.055542    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca" pod "a54825c9-7737-434d-86c3-11c68aa77e3e" (UID: "a54825c9-7737-434d-86c3-11c68aa77e3e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:25:14.055674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.055649    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl" pod "a54825c9-7737-434d-86c3-11c68aa77e3e" (UID: "a54825c9-7737-434d-86c3-11c68aa77e3e"). InnerVolumeSpecName "kube-api-access-hz7dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:25:14.155287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.155246    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz7dl\" (UniqueName: \"kubernetes.io/projected/a54825c9-7737-434d-86c3-11c68aa77e3e-kube-api-access-hz7dl\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:14.155287 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.155283    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a54825c9-7737-434d-86c3-11c68aa77e3e-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:14.201809 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.201767    4528 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:14.202263 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.202252    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:14.202880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.202826    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:14.838567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.838532    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"
Nov 03 06:25:14.838567 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.838552    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh" event={"ID":"a54825c9-7737-434d-86c3-11c68aa77e3e","Type":"ContainerDied","Data":"df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a"}
Nov 03 06:25:14.838789 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:14.838603    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df2dfc75f152bfdbfb8a2319d733e97a387730b08c9d49a4197d081a105b488a"
Nov 03 06:25:15.479026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:15.478995    4528 scope.go:122] "RemoveContainer" containerID="75c15061f7ca7123add8eb7c5add0b9a45a315db702d2ea3ec9857f7c1ef7a49"
Nov 03 06:25:15.488758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:15.488734    4528 scope.go:122] "RemoveContainer" containerID="60fdb4443bc1ef9cc4f89771d770f4f51fb9d817156acd4cb1672d15eda954f4"
Nov 03 06:25:18.737877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:18.737836    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:25:18.771206 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:18.771165    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:25:22.226166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:22.226131    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"]
Nov 03 06:25:22.240500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:22.240454    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-e41e4c4d-42tlh"]
Nov 03 06:25:23.738986 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.738926    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" podUID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" containerName="payload-processing" containerID="cri-o://a914f5ff0133c01441dab7685b0528fb5e09ec69225904764ac5b97f4c88fa8a" gracePeriod=25
Nov 03 06:25:23.748464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.748436    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a54825c9-7737-434d-86c3-11c68aa77e3e" path="/var/lib/kubelet/pods/a54825c9-7737-434d-86c3-11c68aa77e3e/volumes"
Nov 03 06:25:23.771768 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.771711    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" podUID="0a605277-e51d-44dd-922d-56642bd91bac" containerName="payload-pre-processing" containerID="cri-o://7a6cc73bf601dacd0b6c86594f6d91c3a45f9a6b8caee0074043335923ad96eb" gracePeriod=25
Nov 03 06:25:23.898738 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.898710    4528 generic.go:356] "Generic (PLEG): container finished" podID="0a605277-e51d-44dd-922d-56642bd91bac" containerID="7a6cc73bf601dacd0b6c86594f6d91c3a45f9a6b8caee0074043335923ad96eb" exitCode=0
Nov 03 06:25:23.898911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.898786    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" event={"ID":"0a605277-e51d-44dd-922d-56642bd91bac","Type":"ContainerDied","Data":"7a6cc73bf601dacd0b6c86594f6d91c3a45f9a6b8caee0074043335923ad96eb"}
Nov 03 06:25:23.900787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.900766    4528 generic.go:356] "Generic (PLEG): container finished" podID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" containerID="a914f5ff0133c01441dab7685b0528fb5e09ec69225904764ac5b97f4c88fa8a" exitCode=0
Nov 03 06:25:23.900936 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:23.900836    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" event={"ID":"c8d4f73b-d977-45bc-b4b2-d5eea6318274","Type":"ContainerDied","Data":"a914f5ff0133c01441dab7685b0528fb5e09ec69225904764ac5b97f4c88fa8a"}
Nov 03 06:25:24.049237 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.049211    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:25:24.073896 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.073873    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:25:24.195305 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195269    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\") pod \"0a605277-e51d-44dd-922d-56642bd91bac\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") "
Nov 03 06:25:24.195533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195469    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\" (UniqueName: \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\") pod \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") "
Nov 03 06:25:24.195533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195493    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\") pod \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\" (UID: \"c8d4f73b-d977-45bc-b4b2-d5eea6318274\") "
Nov 03 06:25:24.195533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195517    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\" (UniqueName: \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\") pod \"0a605277-e51d-44dd-922d-56642bd91bac\" (UID: \"0a605277-e51d-44dd-922d-56642bd91bac\") "
Nov 03 06:25:24.195770 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195717    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume" pod "0a605277-e51d-44dd-922d-56642bd91bac" (UID: "0a605277-e51d-44dd-922d-56642bd91bac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:25:24.195915 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.195892    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume" pod "c8d4f73b-d977-45bc-b4b2-d5eea6318274" (UID: "c8d4f73b-d977-45bc-b4b2-d5eea6318274"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:25:24.197714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.197685    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm" pod "c8d4f73b-d977-45bc-b4b2-d5eea6318274" (UID: "c8d4f73b-d977-45bc-b4b2-d5eea6318274"). InnerVolumeSpecName "kube-api-access-p5wnm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:25:24.197714 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.197707    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts" pod "0a605277-e51d-44dd-922d-56642bd91bac" (UID: "0a605277-e51d-44dd-922d-56642bd91bac"). InnerVolumeSpecName "kube-api-access-dgdts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:25:24.296767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.296689    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5wnm\" (UniqueName: \"kubernetes.io/projected/c8d4f73b-d977-45bc-b4b2-d5eea6318274-kube-api-access-p5wnm\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:24.296767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.296718    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8d4f73b-d977-45bc-b4b2-d5eea6318274-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:24.296767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.296727    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgdts\" (UniqueName: \"kubernetes.io/projected/0a605277-e51d-44dd-922d-56642bd91bac-kube-api-access-dgdts\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:24.296767 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.296736    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a605277-e51d-44dd-922d-56642bd91bac-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:24.615563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.615516    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:25:24.615779 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.615757    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" containerID="cri-o://32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83" gracePeriod=30
Nov 03 06:25:24.910459 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.910370    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"
Nov 03 06:25:24.910459 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.910397    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv" event={"ID":"c8d4f73b-d977-45bc-b4b2-d5eea6318274","Type":"ContainerDied","Data":"84ee56b2aca11b37b5a5afa86a1e641535b241e74cacd5e987e43d6f95b215a2"}
Nov 03 06:25:24.911024 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.910471    4528 scope.go:122] "RemoveContainer" containerID="a914f5ff0133c01441dab7685b0528fb5e09ec69225904764ac5b97f4c88fa8a"
Nov 03 06:25:24.912143 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.912113    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4" event={"ID":"0a605277-e51d-44dd-922d-56642bd91bac","Type":"ContainerDied","Data":"a5f6987c7434de42813651fd0f7b80129a121477870ce7e0c064c6859b89ad52"}
Nov 03 06:25:24.912274 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.912163    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"
Nov 03 06:25:24.927594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.927569    4528 scope.go:122] "RemoveContainer" containerID="7a6cc73bf601dacd0b6c86594f6d91c3a45f9a6b8caee0074043335923ad96eb"
Nov 03 06:25:24.967411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.967380    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:25:24.972335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:24.972313    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-e41e4c4d-85fff59f55-2dczv"]
Nov 03 06:25:25.005207 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:25.005176    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:25:25.012631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:25.012596    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e41e4c4d-57f4bbb45f-rbsp4"]
Nov 03 06:25:25.748485 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:25.748450    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a605277-e51d-44dd-922d-56642bd91bac" path="/var/lib/kubelet/pods/0a605277-e51d-44dd-922d-56642bd91bac/volumes"
Nov 03 06:25:25.748987 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:25.748966    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" path="/var/lib/kubelet/pods/c8d4f73b-d977-45bc-b4b2-d5eea6318274/volumes"
Nov 03 06:25:29.205487 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.205446    4528 patch_prober.go:29] interesting pod/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 06:25:29.205970 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.205516    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 06:25:29.891142 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.891073    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:29.952463 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.952424    4528 generic.go:356] "Generic (PLEG): container finished" podID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerID="32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83" exitCode=0
Nov 03 06:25:29.952673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.952486    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"
Nov 03 06:25:29.952673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.952526    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerDied","Data":"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"}
Nov 03 06:25:29.952673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.952602    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb" event={"ID":"79c70eb8-b2a7-4758-af1b-fec537d9c18e","Type":"ContainerDied","Data":"0bec62ac0120e04767dfd00d0a541f50ed3ee72656a75a56b207d81f70372cf2"}
Nov 03 06:25:29.952673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.952633    4528 scope.go:122] "RemoveContainer" containerID="32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"
Nov 03 06:25:29.963105 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.963077    4528 scope.go:122] "RemoveContainer" containerID="33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"
Nov 03 06:25:29.972421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.972397    4528 scope.go:122] "RemoveContainer" containerID="32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"
Nov 03 06:25:29.972704 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:29.972685    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83\": container with ID starting with 32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83 not found: ID does not exist" containerID="32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"
Nov 03 06:25:29.972765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.972714    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83"} err="failed to get container status \"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83\": rpc error: code = NotFound desc = could not find container \"32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83\": container with ID starting with 32e22d1231d35fc51227979122f80ca3d9eb155c60a16a28e28eaad20d411d83 not found: ID does not exist"
Nov 03 06:25:29.972765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.972736    4528 scope.go:122] "RemoveContainer" containerID="33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"
Nov 03 06:25:29.973055 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:29.973025    4528 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a\": container with ID starting with 33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a not found: ID does not exist" containerID="33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"
Nov 03 06:25:29.973055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:29.973060    4528 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a"} err="failed to get container status \"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a\": rpc error: code = NotFound desc = could not find container \"33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a\": container with ID starting with 33deec4a520d7bb31ecba2a3631430b01d05a991471a0f2a2a9908ed682b415a not found: ID does not exist"
Nov 03 06:25:30.056817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056760    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056888    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056916    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056937    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056961    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057000 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.056987    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057018    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057105    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057272 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057168    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\") pod \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\" (UID: \"79c70eb8-b2a7-4758-af1b-fec537d9c18e\") "
Nov 03 06:25:30.057442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057243    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:25:30.057442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057320    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:25:30.057559 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057433    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:25:30.057636 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057594    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:25:30.057712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057635    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:25:30.057712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057660    4528 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-data\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.057712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057684    4528 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-credential-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.057712 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.057700    4528 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-certs\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.059623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.059598    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 06:25:30.059713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.059612    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "kube-api-access-fp46j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:25:30.059853 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.059830    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 06:25:30.059900 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.059876    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token" pod "79c70eb8-b2a7-4758-af1b-fec537d9c18e" (UID: "79c70eb8-b2a7-4758-af1b-fec537d9c18e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158542    4528 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-envoy\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158575    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp46j\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-kube-api-access-fp46j\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158586    4528 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79c70eb8-b2a7-4758-af1b-fec537d9c18e-workload-socket\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158597    4528 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istiod-ca-cert\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158606    4528 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-token\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.158626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.158613    4528 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79c70eb8-b2a7-4758-af1b-fec537d9c18e-istio-podinfo\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:25:30.297751 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.297712    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:25:30.309784 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:30.309750    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-e41e4c4d-openshift-default-58cbc9d745-tntdb"]
Nov 03 06:25:31.749492 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.749453    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" path="/var/lib/kubelet/pods/79c70eb8-b2a7-4758-af1b-fec537d9c18e/volumes"
Nov 03 06:25:31.965201 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965162    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:25:31.965603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965590    4528 cpu_manager.go:414] "containerMap: removing container" podUID="a54825c9-7737-434d-86c3-11c68aa77e3e" containerName="revoke-keys"
Nov 03 06:25:31.965603 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965603    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a54825c9-7737-434d-86c3-11c68aa77e3e" containerName="revoke-keys"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965615    4528 cpu_manager.go:414] "containerMap: removing container" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965620    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965628    4528 cpu_manager.go:414] "containerMap: removing container" podUID="0a605277-e51d-44dd-922d-56642bd91bac" containerName="payload-pre-processing"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965633    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a605277-e51d-44dd-922d-56642bd91bac" containerName="payload-pre-processing"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965649    4528 cpu_manager.go:414] "containerMap: removing container" podUID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" containerName="payload-processing"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965653    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" containerName="payload-processing"
Nov 03 06:25:31.965742 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965738    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8d4f73b-d977-45bc-b4b2-d5eea6318274" containerName="payload-processing"
Nov 03 06:25:31.966131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965749    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:31.966131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965760    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="a54825c9-7737-434d-86c3-11c68aa77e3e" containerName="revoke-keys"
Nov 03 06:25:31.966131 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.965767    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a605277-e51d-44dd-922d-56642bd91bac" containerName="payload-pre-processing"
Nov 03 06:25:31.969138 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.969118    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:31.987472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.987443    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:25:31.994683 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.994658    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-5797b0\""
Nov 03 06:25:31.994917 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:31.994898    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-5797b0-dockercfg-8rvpj\""
Nov 03 06:25:32.073438 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.073394    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:25:32.074205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.074168    4528 cpu_manager.go:414] "containerMap: removing container" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:32.074205 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.074195    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:32.074376 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.074337    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="79c70eb8-b2a7-4758-af1b-fec537d9c18e" containerName="istio-proxy"
Nov 03 06:25:32.077864 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.077830    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28x7w\" (UniqueName: \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.081019 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.077951    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.091041 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.091008    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.101988 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.101960    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:25:32.185985 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.185944    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28x7w\" (UniqueName: \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.186187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.186020    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6z56\" (UniqueName: \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.186187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.186050    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.186319 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.186175    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.186876 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.186856    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.211423 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.211400    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28x7w\" (UniqueName: \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\") pod \"payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.281934 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.281898    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:32.287387 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.287359    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6z56\" (UniqueName: \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.287521 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.287487    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.288160 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.288141    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.310550 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.310519    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6z56\" (UniqueName: \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\") pod \"payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.412456 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.412414    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.581813 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.581767    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:25:32.582758 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:25:32.582732    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2f08261_2134_4912_aa40_ac07ceced003.slice/crio-79c4be97776f34c116343d48a40923676170526bb8583eb4683ab21246310e1d WatchSource:0}: Error finding container 79c4be97776f34c116343d48a40923676170526bb8583eb4683ab21246310e1d: Status 404 returned error can't find the container with id 79c4be97776f34c116343d48a40923676170526bb8583eb4683ab21246310e1d
Nov 03 06:25:32.712171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.712138    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:25:32.712728 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:25:32.712701    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4539cd20_f4c1_4891_8cb2_a1746a09e398.slice/crio-fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66 WatchSource:0}: Error finding container fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66: Status 404 returned error can't find the container with id fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66
Nov 03 06:25:32.975577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.975467    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" event={"ID":"4539cd20-f4c1-4891-8cb2-a1746a09e398","Type":"ContainerStarted","Data":"4d1e5fd1510c5445bc3eecae4c5e0648752cc2fbe3062e421cd63b3573a79931"}
Nov 03 06:25:32.975577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.975540    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:32.975577 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.975566    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" event={"ID":"4539cd20-f4c1-4891-8cb2-a1746a09e398","Type":"ContainerStarted","Data":"fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66"}
Nov 03 06:25:32.977302 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.977271    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" event={"ID":"f2f08261-2134-4912-aa40-ac07ceced003","Type":"ContainerStarted","Data":"4de98aafcc2b12cb83a182192a256ea23bd08221107f9d83760cbd5b8193aead"}
Nov 03 06:25:32.977468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.977321    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" event={"ID":"f2f08261-2134-4912-aa40-ac07ceced003","Type":"ContainerStarted","Data":"79c4be97776f34c116343d48a40923676170526bb8583eb4683ab21246310e1d"}
Nov 03 06:25:32.977468 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:32.977344    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:25:33.016781 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:33.016720    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" podStartSLOduration=1.016700746 podStartE2EDuration="1.016700746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:25:33.016479433 +0000 UTC m=+1821.427876509" watchObservedRunningTime="2026-11-03 06:25:33.016700746 +0000 UTC m=+1821.428097824"
Nov 03 06:25:33.056362 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:33.056294    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" podStartSLOduration=2.056275456 podStartE2EDuration="2.056275456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:25:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:25:33.055048398 +0000 UTC m=+1821.466445476" watchObservedRunningTime="2026-11-03 06:25:33.056275456 +0000 UTC m=+1821.467672535"
Nov 03 06:25:43.694000 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:25:43.693957    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:25:43.987053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:43.986965    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:25:43.987053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:25:43.987046    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:26:10.027265 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:10.027225    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"]
Nov 03 06:26:10.038533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:10.038493    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rwzl6"]
Nov 03 06:26:11.748435 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:11.748403    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f4f6270-1d1f-46ae-82a8-d5d1c3426743" path="/var/lib/kubelet/pods/0f4f6270-1d1f-46ae-82a8-d5d1c3426743/volumes"
Nov 03 06:26:13.693489 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:26:13.693448    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:26:15.522030 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:15.521998    4528 scope.go:122] "RemoveContainer" containerID="8533d0b74fe72218a7e72a84548bfa27c46ac0e2f56763703a507c840ccb9428"
Nov 03 06:26:31.033240 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:31.033204    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"]
Nov 03 06:26:31.042881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:31.042847    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-pd686"]
Nov 03 06:26:31.747846 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:26:31.747817    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dbada2d-7933-46a8-ae3e-87e6355f8eb8" path="/var/lib/kubelet/pods/4dbada2d-7933-46a8-ae3e-87e6355f8eb8/volumes"
Nov 03 06:26:43.691895 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:26:43.691853    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:27:04.903039 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.903003    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"]
Nov 03 06:27:04.906819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.906783    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:04.924914 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.924886    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"]
Nov 03 06:27:04.932375 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.932344    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:27:04.932533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.932391    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:27:04.932533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.932438    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:27:04.932533 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:04.932482    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6p27d\""
Nov 03 06:27:05.067115 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.067081    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.067309 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.067131    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.067309 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.067265    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf8c2\" (UniqueName: \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.168642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.168549    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf8c2\" (UniqueName: \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.168642 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.168619    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.168911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.168652    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.169472 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.169442    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.171134 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.171112    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.191106 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.191085    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf8c2\" (UniqueName: \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\") pod \"maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.219933 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.219909    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:05.518275 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.518250    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"]
Nov 03 06:27:05.519564 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:27:05.519539    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d91ee6a_1b98_4d8c_b22f_5ce7dcffffcd.slice/crio-54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0 WatchSource:0}: Error finding container 54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0: Status 404 returned error can't find the container with id 54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0
Nov 03 06:27:05.556232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:05.556203    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd" event={"ID":"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd","Type":"ContainerStarted","Data":"54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0"}
Nov 03 06:27:06.563827 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:06.563724    4528 generic.go:356] "Generic (PLEG): container finished" podID="9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" containerID="7c16bc7348f9c46966d931c01afde7e04a3fee8c6c8b918f3997dd09639f7a8a" exitCode=0
Nov 03 06:27:06.564233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:06.563845    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd" event={"ID":"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd","Type":"ContainerDied","Data":"7c16bc7348f9c46966d931c01afde7e04a3fee8c6c8b918f3997dd09639f7a8a"}
Nov 03 06:27:07.710407 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.710380    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:07.897774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.897681    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\" (UniqueName: \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\") pod \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") "
Nov 03 06:27:07.897774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.897764    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\") pod \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") "
Nov 03 06:27:07.898053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.897889    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\") pod \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\" (UID: \"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd\") "
Nov 03 06:27:07.898187 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.898162    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca" pod "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" (UID: "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:27:07.899969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.899945    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca" pod "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" (UID: "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:27:07.900069 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.899999    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2" pod "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" (UID: "9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd"). InnerVolumeSpecName "kube-api-access-kf8c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:27:07.999452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.999416    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:07.999452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.999448    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:07.999452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:07.999459    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf8c2\" (UniqueName: \"kubernetes.io/projected/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd-kube-api-access-kf8c2\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:08.577517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:08.577482    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"
Nov 03 06:27:08.577676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:08.577522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd" event={"ID":"9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd","Type":"ContainerDied","Data":"54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0"}
Nov 03 06:27:08.577676 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:08.577583    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54022ee14be7f6a2f1a05a008c54085453a9be4f25f0efc89051b4330c85b4a0"
Nov 03 06:27:13.692153 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:27:13.692116    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:27:15.364125 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:15.364091    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:27:15.397640 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:15.397602    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:27:15.565409 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:15.565379    4528 scope.go:122] "RemoveContainer" containerID="446b4670b158c920a884aa8bfa57fd0b80c65261bf95a3a6314db76da3bcf364"
Nov 03 06:27:20.364569 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.364495    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" podUID="4539cd20-f4c1-4891-8cb2-a1746a09e398" containerName="payload-processing" containerID="cri-o://4d1e5fd1510c5445bc3eecae4c5e0648752cc2fbe3062e421cd63b3573a79931" gracePeriod=25
Nov 03 06:27:20.405581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.405529    4528 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" podUID="f2f08261-2134-4912-aa40-ac07ceced003" containerName="payload-pre-processing" containerID="cri-o://4de98aafcc2b12cb83a182192a256ea23bd08221107f9d83760cbd5b8193aead" gracePeriod=25
Nov 03 06:27:20.658418 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.658388    4528 generic.go:356] "Generic (PLEG): container finished" podID="4539cd20-f4c1-4891-8cb2-a1746a09e398" containerID="4d1e5fd1510c5445bc3eecae4c5e0648752cc2fbe3062e421cd63b3573a79931" exitCode=0
Nov 03 06:27:20.658625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.658470    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" event={"ID":"4539cd20-f4c1-4891-8cb2-a1746a09e398","Type":"ContainerDied","Data":"4d1e5fd1510c5445bc3eecae4c5e0648752cc2fbe3062e421cd63b3573a79931"}
Nov 03 06:27:20.658625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.658577    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t" event={"ID":"4539cd20-f4c1-4891-8cb2-a1746a09e398","Type":"ContainerDied","Data":"fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66"}
Nov 03 06:27:20.658625 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.658606    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc15d1c0acff26b03ad08759014c3579a9b38efa0d6b88b374ef563722ea5d66"
Nov 03 06:27:20.660568 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.660545    4528 generic.go:356] "Generic (PLEG): container finished" podID="f2f08261-2134-4912-aa40-ac07ceced003" containerID="4de98aafcc2b12cb83a182192a256ea23bd08221107f9d83760cbd5b8193aead" exitCode=0
Nov 03 06:27:20.660674 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.660576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" event={"ID":"f2f08261-2134-4912-aa40-ac07ceced003","Type":"ContainerDied","Data":"4de98aafcc2b12cb83a182192a256ea23bd08221107f9d83760cbd5b8193aead"}
Nov 03 06:27:20.671297 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.671274    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:27:20.714663 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.714629    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:27:20.727877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.727119    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\") pod \"4539cd20-f4c1-4891-8cb2-a1746a09e398\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") "
Nov 03 06:27:20.727877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.727353    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\" (UniqueName: \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\") pod \"4539cd20-f4c1-4891-8cb2-a1746a09e398\" (UID: \"4539cd20-f4c1-4891-8cb2-a1746a09e398\") "
Nov 03 06:27:20.727877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.727492    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume" pod "4539cd20-f4c1-4891-8cb2-a1746a09e398" (UID: "4539cd20-f4c1-4891-8cb2-a1746a09e398"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:27:20.727877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.727727    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4539cd20-f4c1-4891-8cb2-a1746a09e398-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:20.729580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.729551    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56" pod "4539cd20-f4c1-4891-8cb2-a1746a09e398" (UID: "4539cd20-f4c1-4891-8cb2-a1746a09e398"). InnerVolumeSpecName "kube-api-access-g6z56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:27:20.828114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.828075    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\") pod \"f2f08261-2134-4912-aa40-ac07ceced003\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") "
Nov 03 06:27:20.828329 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.828204    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\" (UniqueName: \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\") pod \"f2f08261-2134-4912-aa40-ac07ceced003\" (UID: \"f2f08261-2134-4912-aa40-ac07ceced003\") "
Nov 03 06:27:20.828536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.828517    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6z56\" (UniqueName: \"kubernetes.io/projected/4539cd20-f4c1-4891-8cb2-a1746a09e398-kube-api-access-g6z56\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:20.828600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.828565    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume" pod "f2f08261-2134-4912-aa40-ac07ceced003" (UID: "f2f08261-2134-4912-aa40-ac07ceced003"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:27:20.830289 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.830269    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w" pod "f2f08261-2134-4912-aa40-ac07ceced003" (UID: "f2f08261-2134-4912-aa40-ac07ceced003"). InnerVolumeSpecName "kube-api-access-28x7w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:27:20.930162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.930066    4528 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2f08261-2134-4912-aa40-ac07ceced003-plugins-config-volume\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:20.930162 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:20.930108    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28x7w\" (UniqueName: \"kubernetes.io/projected/f2f08261-2134-4912-aa40-ac07ceced003-kube-api-access-28x7w\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:21.667969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.667887    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"
Nov 03 06:27:21.667969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.667905    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6" event={"ID":"f2f08261-2134-4912-aa40-ac07ceced003","Type":"ContainerDied","Data":"79c4be97776f34c116343d48a40923676170526bb8583eb4683ab21246310e1d"}
Nov 03 06:27:21.667969 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.667967    4528 scope.go:122] "RemoveContainer" containerID="4de98aafcc2b12cb83a182192a256ea23bd08221107f9d83760cbd5b8193aead"
Nov 03 06:27:21.668469 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.668160    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"
Nov 03 06:27:21.718386 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.718355    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:27:21.729728 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.729695    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-5797b0-6f799c56cf-ktlp6"]
Nov 03 06:27:21.748473 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.748440    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2f08261-2134-4912-aa40-ac07ceced003" path="/var/lib/kubelet/pods/f2f08261-2134-4912-aa40-ac07ceced003/volumes"
Nov 03 06:27:21.754261 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.754233    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:27:21.760489 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:21.760460    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-5797b0-74d6cf959f-5842t"]
Nov 03 06:27:22.729866 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:22.729823    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"]
Nov 03 06:27:22.739903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:22.739873    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-authcfg-http-5797b0-r9sxd"]
Nov 03 06:27:23.747359 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:23.747286    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4539cd20-f4c1-4891-8cb2-a1746a09e398" path="/var/lib/kubelet/pods/4539cd20-f4c1-4891-8cb2-a1746a09e398/volumes"
Nov 03 06:27:23.747739 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:23.747720    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" path="/var/lib/kubelet/pods/9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd/volumes"
Nov 03 06:27:37.028910 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:37.028871    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"]
Nov 03 06:27:37.039765 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:37.039729    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a651a15-gcbhm"]
Nov 03 06:27:37.747699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:37.747666    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e3a11a5-ece0-473f-9191-0e25881b179f" path="/var/lib/kubelet/pods/2e3a11a5-ece0-473f-9191-0e25881b179f/volumes"
Nov 03 06:27:43.662036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:43.661978    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:27:43.662687 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:43.662630    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:27:43.858818 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:27:43.845985    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:27:47.377477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377440    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"]
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377914    4528 cpu_manager.go:414] "containerMap: removing container" podUID="4539cd20-f4c1-4891-8cb2-a1746a09e398" containerName="payload-processing"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377926    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4539cd20-f4c1-4891-8cb2-a1746a09e398" containerName="payload-processing"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377941    4528 cpu_manager.go:414] "containerMap: removing container" podUID="f2f08261-2134-4912-aa40-ac07ceced003" containerName="payload-pre-processing"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377946    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2f08261-2134-4912-aa40-ac07ceced003" containerName="payload-pre-processing"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377965    4528 cpu_manager.go:414] "containerMap: removing container" podUID="9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" containerName="revoke-keys"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.377970    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" containerName="revoke-keys"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.378042    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d91ee6a-1b98-4d8c-b22f-5ce7dcffffcd" containerName="revoke-keys"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.378055    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="4539cd20-f4c1-4891-8cb2-a1746a09e398" containerName="payload-processing"
Nov 03 06:27:47.378074 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.378063    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2f08261-2134-4912-aa40-ac07ceced003" containerName="payload-pre-processing"
Nov 03 06:27:47.381562 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.381543    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.407836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.407772    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:27:47.407836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.407800    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6p27d\""
Nov 03 06:27:47.407836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.407802    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:27:47.408147 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.408079    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:27:47.411112 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.411089    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"]
Nov 03 06:27:47.538018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.537969    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kn9m\" (UniqueName: \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.538196 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.538027    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.538244 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.538198    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.638785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.638676    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kn9m\" (UniqueName: \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.638785 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.638739    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.639058 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.638902    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.639452 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.639431    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.641233 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.641212    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.661536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.661494    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kn9m\" (UniqueName: \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\") pod \"maas-api-revoke-sub-f03629db-9wwm9\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.694168 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.694134    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:47.994161 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:47.994135    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"]
Nov 03 06:27:47.995691 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:27:47.995654    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd4718a_2e42_4f41_abbe_fe8ed0338d15.slice/crio-90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9 WatchSource:0}: Error finding container 90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9: Status 404 returned error can't find the container with id 90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9
Nov 03 06:27:48.972480 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:48.972443    4528 generic.go:356] "Generic (PLEG): container finished" podID="5cd4718a-2e42-4f41-abbe-fe8ed0338d15" containerID="0e70d2aa7bfe9c33abc6108f40a9ea4149b9ab27f14a8a23ed0783f1f4fb45cc" exitCode=0
Nov 03 06:27:48.972954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:48.972522    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9" event={"ID":"5cd4718a-2e42-4f41-abbe-fe8ed0338d15","Type":"ContainerDied","Data":"0e70d2aa7bfe9c33abc6108f40a9ea4149b9ab27f14a8a23ed0783f1f4fb45cc"}
Nov 03 06:27:48.972954 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:48.972576    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9" event={"ID":"5cd4718a-2e42-4f41-abbe-fe8ed0338d15","Type":"ContainerStarted","Data":"90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9"}
Nov 03 06:27:50.120688 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.120660    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:50.264685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.264581    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\") pod \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") "
Nov 03 06:27:50.264685 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.264630    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\") pod \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") "
Nov 03 06:27:50.264982 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.264705    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\" (UniqueName: \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\") pod \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\" (UID: \"5cd4718a-2e42-4f41-abbe-fe8ed0338d15\") "
Nov 03 06:27:50.265081 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.265052    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca" pod "5cd4718a-2e42-4f41-abbe-fe8ed0338d15" (UID: "5cd4718a-2e42-4f41-abbe-fe8ed0338d15"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:27:50.266786 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.266765    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m" pod "5cd4718a-2e42-4f41-abbe-fe8ed0338d15" (UID: "5cd4718a-2e42-4f41-abbe-fe8ed0338d15"). InnerVolumeSpecName "kube-api-access-2kn9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:27:50.266884 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.266862    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca" pod "5cd4718a-2e42-4f41-abbe-fe8ed0338d15" (UID: "5cd4718a-2e42-4f41-abbe-fe8ed0338d15"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:27:50.366054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.366012    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kn9m\" (UniqueName: \"kubernetes.io/projected/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-kube-api-access-2kn9m\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:50.366054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.366046    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:50.366054 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.366059    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5cd4718a-2e42-4f41-abbe-fe8ed0338d15-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:27:50.986198 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.986162    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9"
Nov 03 06:27:50.986198 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.986191    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f03629db-9wwm9" event={"ID":"5cd4718a-2e42-4f41-abbe-fe8ed0338d15","Type":"ContainerDied","Data":"90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9"}
Nov 03 06:27:50.986465 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:27:50.986242    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a3870872d2473429cc5e5588160dcaf4f02c0794b14cae182b25c48a5dd7f9"
Nov 03 06:28:04.475731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.475691    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"]
Nov 03 06:28:04.476461 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.476438    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5cd4718a-2e42-4f41-abbe-fe8ed0338d15" containerName="revoke-keys"
Nov 03 06:28:04.476511 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.476464    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd4718a-2e42-4f41-abbe-fe8ed0338d15" containerName="revoke-keys"
Nov 03 06:28:04.476623 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.476612    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd4718a-2e42-4f41-abbe-fe8ed0338d15" containerName="revoke-keys"
Nov 03 06:28:04.480605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.480584    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.496659 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.496628    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"]
Nov 03 06:28:04.504419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.504381    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:28:04.504419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.504405    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:28:04.504419 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.504426    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:28:04.506251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.506223    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6p27d\""
Nov 03 06:28:04.556337 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.556284    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.556510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.556389    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfgr\" (UniqueName: \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.556510 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.556458    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.657626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.657585    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfgr\" (UniqueName: \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.657860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.657678    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.657860 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.657742    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.658451 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.658424    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.660008 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.659990    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.682497 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.682463    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfgr\" (UniqueName: \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\") pod \"maas-api-revoke-sub-0165d7d2-fjkxx\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:04.794699 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:04.794597    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:05.091646 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:05.091615    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"]
Nov 03 06:28:05.092685 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:28:05.092661    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce7047e_d494_4306_9641_984dd65c86ef.slice/crio-35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940 WatchSource:0}: Error finding container 35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940: Status 404 returned error can't find the container with id 35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940
Nov 03 06:28:05.156011 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:05.155974    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx" event={"ID":"5ce7047e-d494-4306-9641-984dd65c86ef","Type":"ContainerStarted","Data":"35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940"}
Nov 03 06:28:06.163166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:06.163084    4528 generic.go:356] "Generic (PLEG): container finished" podID="5ce7047e-d494-4306-9641-984dd65c86ef" containerID="fe4030913a068791e1045e75e5604cbb5a1a3d26c4f96491ea85d7570cb5243b" exitCode=0
Nov 03 06:28:06.163526 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:06.163167    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx" event={"ID":"5ce7047e-d494-4306-9641-984dd65c86ef","Type":"ContainerDied","Data":"fe4030913a068791e1045e75e5604cbb5a1a3d26c4f96491ea85d7570cb5243b"}
Nov 03 06:28:07.309609 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.309582    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:07.387788 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.387752    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\") pod \"5ce7047e-d494-4306-9641-984dd65c86ef\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") "
Nov 03 06:28:07.388018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.387834    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\") pod \"5ce7047e-d494-4306-9641-984dd65c86ef\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") "
Nov 03 06:28:07.388018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.387932    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\" (UniqueName: \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\") pod \"5ce7047e-d494-4306-9641-984dd65c86ef\" (UID: \"5ce7047e-d494-4306-9641-984dd65c86ef\") "
Nov 03 06:28:07.388183 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.388145    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca" pod "5ce7047e-d494-4306-9641-984dd65c86ef" (UID: "5ce7047e-d494-4306-9641-984dd65c86ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:28:07.388353 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.388332    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:07.390088 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.390060    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca" pod "5ce7047e-d494-4306-9641-984dd65c86ef" (UID: "5ce7047e-d494-4306-9641-984dd65c86ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:28:07.390197 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.390124    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr" pod "5ce7047e-d494-4306-9641-984dd65c86ef" (UID: "5ce7047e-d494-4306-9641-984dd65c86ef"). InnerVolumeSpecName "kube-api-access-jmfgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:28:07.489563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.489464    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5ce7047e-d494-4306-9641-984dd65c86ef-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:07.489563 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:07.489509    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmfgr\" (UniqueName: \"kubernetes.io/projected/5ce7047e-d494-4306-9641-984dd65c86ef-kube-api-access-jmfgr\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:08.176341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:08.176308    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx"
Nov 03 06:28:08.176341 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:08.176333    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0165d7d2-fjkxx" event={"ID":"5ce7047e-d494-4306-9641-984dd65c86ef","Type":"ContainerDied","Data":"35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940"}
Nov 03 06:28:08.176566 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:08.176382    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35a896fddf543dc82578c0da5bbc1a95fb3e07718172ce4e65a034249e5e5940"
Nov 03 06:28:13.692346 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:28:13.692279    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:28:15.529477 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.529436    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"]
Nov 03 06:28:15.530177 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.530156    4528 cpu_manager.go:414] "containerMap: removing container" podUID="5ce7047e-d494-4306-9641-984dd65c86ef" containerName="revoke-keys"
Nov 03 06:28:15.530255 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.530183    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce7047e-d494-4306-9641-984dd65c86ef" containerName="revoke-keys"
Nov 03 06:28:15.530335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.530309    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce7047e-d494-4306-9641-984dd65c86ef" containerName="revoke-keys"
Nov 03 06:28:15.534564 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.534541    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.554330 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.554303    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"]
Nov 03 06:28:15.554554 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.554532    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:28:15.561993 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.561966    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 03 06:28:15.562126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.562043    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6p27d\""
Nov 03 06:28:15.562126 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.562055    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:28:15.577476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.577449    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.577655 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.577628    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.577705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.577690    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg4ng\" (UniqueName: \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.590671 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.590639    4528 scope.go:122] "RemoveContainer" containerID="95b4f2a4940e26ab01088ce8c889afc2499b577cf6790008b96012de943d59d5"
Nov 03 06:28:15.679239 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.679200    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.679416 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.679312    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg4ng\" (UniqueName: \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.679507 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.679431    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.679911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.679892    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.681758 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.681735    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.703920 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.703896    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg4ng\" (UniqueName: \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\") pod \"maas-api-revoke-sub-6b19338f-dnnsp\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:15.851991 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:15.851956    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:16.165457 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:16.165429    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"]
Nov 03 06:28:16.166082 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:28:16.166057    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991ade28_f156_4851_ae9e_5924b8af073d.slice/crio-4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3 WatchSource:0}: Error finding container 4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3: Status 404 returned error can't find the container with id 4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3
Nov 03 06:28:16.235885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:16.235838    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp" event={"ID":"991ade28-f156-4851-ae9e-5924b8af073d","Type":"ContainerStarted","Data":"4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3"}
Nov 03 06:28:17.239015 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:17.238978    4528 generic.go:356] "Generic (PLEG): container finished" podID="991ade28-f156-4851-ae9e-5924b8af073d" containerID="05289de602d7829e89a6b4b0a617e48a184ac51092296209a36ac6f42acf8f1a" exitCode=0
Nov 03 06:28:17.239422 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:17.239060    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp" event={"ID":"991ade28-f156-4851-ae9e-5924b8af073d","Type":"ContainerDied","Data":"05289de602d7829e89a6b4b0a617e48a184ac51092296209a36ac6f42acf8f1a"}
Nov 03 06:28:18.378965 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.378936    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:18.513841 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.510609    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\") pod \"991ade28-f156-4851-ae9e-5924b8af073d\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") "
Nov 03 06:28:18.513841 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.510766    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\") pod \"991ade28-f156-4851-ae9e-5924b8af073d\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") "
Nov 03 06:28:18.513841 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.510902    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\" (UniqueName: \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\") pod \"991ade28-f156-4851-ae9e-5924b8af073d\" (UID: \"991ade28-f156-4851-ae9e-5924b8af073d\") "
Nov 03 06:28:18.515232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.514620    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng" pod "991ade28-f156-4851-ae9e-5924b8af073d" (UID: "991ade28-f156-4851-ae9e-5924b8af073d"). InnerVolumeSpecName "kube-api-access-zg4ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:28:18.515232 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.514964    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca" pod "991ade28-f156-4851-ae9e-5924b8af073d" (UID: "991ade28-f156-4851-ae9e-5924b8af073d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:28:18.518509 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.517377    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca" pod "991ade28-f156-4851-ae9e-5924b8af073d" (UID: "991ade28-f156-4851-ae9e-5924b8af073d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 06:28:18.612890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.612855    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-xks-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:18.612890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.612885    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg4ng\" (UniqueName: \"kubernetes.io/projected/991ade28-f156-4851-ae9e-5924b8af073d-kube-api-access-zg4ng\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:18.612890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:18.612895    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/991ade28-f156-4851-ae9e-5924b8af073d-maas-api-openshift-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:28:19.252393 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:19.252363    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp"
Nov 03 06:28:19.252581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:19.252393    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b19338f-dnnsp" event={"ID":"991ade28-f156-4851-ae9e-5924b8af073d","Type":"ContainerDied","Data":"4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3"}
Nov 03 06:28:19.252581 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:19.252449    4528 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4cd4b1d3bfc8431b7d712a325d69723cd1be512aca0e9caf532b1ce048a9a3a3"
Nov 03 06:28:28.667950 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:28.667902    4528 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-778b7b5c47-c4nqk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 06:28:28.668657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:28.668623    4528 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-778b7b5c47-c4nqk" podUID="e8c4a3ba-5a15-40ac-823e-795c43744b3c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.28:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 06:28:37.500669 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:37.500636    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f7fb74d9c-c8g4b_0c7a3d80-23c6-435f-8847-98330817b665/manager/0.log"
Nov 03 06:28:40.225218 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:40.225183    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjkxx_5ce7047e-d494-4306-9641-984dd65c86ef/revoke-keys/0.log"
Nov 03 06:28:40.389877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:40.389844    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fn8hg_96928d09-a89d-444c-a690-70aa5155735f/revoke-keys/0.log"
Nov 03 06:28:40.555503 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:40.555477    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-qjlx2_807febcc-ec5a-477c-a56f-f45956e55c40/revoke-keys/0.log"
Nov 03 06:28:40.737703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:40.737671    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dnnsp_991ade28-f156-4851-ae9e-5924b8af073d/revoke-keys/0.log"
Nov 03 06:28:40.921935 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:40.921849    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-af191ac6-mvhpp_41dec697-134c-40b5-99b5-77846f9975f5/revoke-keys/0.log"
Nov 03 06:28:41.112096 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:41.112068    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9wwm9_5cd4718a-2e42-4f41-abbe-fe8ed0338d15/revoke-keys/0.log"
Nov 03 06:28:42.992830 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:42.992782    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xjzbk_40c16b20-952a-40a3-892b-06a6dbfd0bb5/manager/0.log"
Nov 03 06:28:43.182350 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:43.182301    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cmx7d_491fbb46-3ea5-459b-be62-ab56797e6972/manager/0.log"
Nov 03 06:28:43.376018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:43.375989    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb997_b7074acb-4264-4990-b61e-f30b57821f6d/kuadrant-console-plugin/0.log"
Nov 03 06:28:43.576413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:43.576376    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g6r8h_dd057882-81dc-4bd5-b249-4ea76ecbe396/manager/0.log"
Nov 03 06:28:43.692709 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:28:43.692621    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:28:44.737378 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:44.737347    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485wjs6f_aba44b32-7a71-462f-a683-7d6e39f140ac/istio-proxy/0.log"
Nov 03 06:28:45.679602 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:45.679571    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-8mbmr_11852422-e9b2-4cc4-a315-644dd38888f9/istio-proxy/0.log"
Nov 03 06:28:46.075100 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:46.075070    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/1.log"
Nov 03 06:28:46.860774 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:46.860740    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-q2zk7_138857e2-d465-4298-a8af-7ebfbab95b86/istio-proxy/0.log"
Nov 03 06:28:47.257036 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:47.256952    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-zw8r9_bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb/istio-proxy/0.log"
Nov 03 06:28:47.444280 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:47.444239    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4_63f8190f-d2db-4674-b2dc-7894853d97ec/istio-proxy/0.log"
Nov 03 06:28:47.637822 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:47.637763    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9m2mq_1c767011-9f84-4a57-9d95-727814cc25e4/payload-pre-processing/0.log"
Nov 03 06:28:47.805535 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:47.805507    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:28:48.435371 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:48.435336    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/0.log"
Nov 03 06:28:48.975171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:48.975140    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv_15775b64-f27a-4c4a-8eeb-3455396727c9/main/0.log"
Nov 03 06:28:49.379455 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:49.379421    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b8h79_6a36b5bf-8092-44f9-9317-d1d1120fe40d/main/0.log"
Nov 03 06:28:49.554464 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:49.554433    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dxpcj_3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5/main/0.log"
Nov 03 06:28:50.106506 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:50.106479    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4btct_68da6dd2-14bd-413e-ab3b-0118d96be1b0/main/0.log"
Nov 03 06:28:54.030983 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:54.030883    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"]
Nov 03 06:28:54.042238 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:54.042203    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-qjlx2"]
Nov 03 06:28:55.750356 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:28:55.750312    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="807febcc-ec5a-477c-a56f-f45956e55c40" path="/var/lib/kubelet/pods/807febcc-ec5a-477c-a56f-f45956e55c40/volumes"
Nov 03 06:29:00.126925 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:00.126892    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-7f7fb74d9c-c8g4b_0c7a3d80-23c6-435f-8847-98330817b665/manager/0.log"
Nov 03 06:29:02.741189 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:02.741152    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0165d7d2-fjkxx_5ce7047e-d494-4306-9641-984dd65c86ef/revoke-keys/0.log"
Nov 03 06:29:02.926047 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:02.926006    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-fn8hg_96928d09-a89d-444c-a690-70aa5155735f/revoke-keys/0.log"
Nov 03 06:29:03.124621 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:03.124574    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-6b19338f-dnnsp_991ade28-f156-4851-ae9e-5924b8af073d/revoke-keys/0.log"
Nov 03 06:29:03.326340 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:03.326311    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-af191ac6-mvhpp_41dec697-134c-40b5-99b5-77846f9975f5/revoke-keys/0.log"
Nov 03 06:29:03.514006 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:03.513900    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-f03629db-9wwm9_5cd4718a-2e42-4f41-abbe-fe8ed0338d15/revoke-keys/0.log"
Nov 03 06:29:05.064661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:05.064598    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xjzbk_40c16b20-952a-40a3-892b-06a6dbfd0bb5/manager/0.log"
Nov 03 06:29:05.282333 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:05.282299    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cmx7d_491fbb46-3ea5-459b-be62-ab56797e6972/manager/0.log"
Nov 03 06:29:05.477346 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:05.477273    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb997_b7074acb-4264-4990-b61e-f30b57821f6d/kuadrant-console-plugin/0.log"
Nov 03 06:29:05.649745 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:05.649700    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g6r8h_dd057882-81dc-4bd5-b249-4ea76ecbe396/manager/0.log"
Nov 03 06:29:06.662412 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:06.662383    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485wjs6f_aba44b32-7a71-462f-a683-7d6e39f140ac/istio-proxy/0.log"
Nov 03 06:29:07.657351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:07.657322    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-8mbmr_11852422-e9b2-4cc4-a315-644dd38888f9/istio-proxy/0.log"
Nov 03 06:29:08.074296 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:08.074262    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/1.log"
Nov 03 06:29:08.868335 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:08.868301    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-q2zk7_138857e2-d465-4298-a8af-7ebfbab95b86/istio-proxy/0.log"
Nov 03 06:29:09.266911 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:09.266829    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-zw8r9_bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb/istio-proxy/0.log"
Nov 03 06:29:09.461366 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:09.461331    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4_63f8190f-d2db-4674-b2dc-7894853d97ec/istio-proxy/0.log"
Nov 03 06:29:09.646903 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:09.646869    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9m2mq_1c767011-9f84-4a57-9d95-727814cc25e4/payload-pre-processing/0.log"
Nov 03 06:29:09.828424 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:09.828395    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:29:10.532022 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:10.531995    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-9xppv_15775b64-f27a-4c4a-8eeb-3455396727c9/main/0.log"
Nov 03 06:29:10.934580 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:10.934552    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-b8h79_6a36b5bf-8092-44f9-9317-d1d1120fe40d/main/0.log"
Nov 03 06:29:11.030476 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:11.030441    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"]
Nov 03 06:29:11.041286 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:11.041254    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-fn8hg"]
Nov 03 06:29:11.115251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:11.115220    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-dxpcj_3e692fdf-8ed2-4ccc-8cb3-86f39898b3d5/main/0.log"
Nov 03 06:29:11.700943 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:11.700912    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-4btct_68da6dd2-14bd-413e-ab3b-0118d96be1b0/main/0.log"
Nov 03 06:29:11.748283 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:11.748253    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96928d09-a89d-444c-a690-70aa5155735f" path="/var/lib/kubelet/pods/96928d09-a89d-444c-a690-70aa5155735f/volumes"
Nov 03 06:29:13.692138 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:29:13.692102    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:29:15.628222 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:15.628191    4528 scope.go:122] "RemoveContainer" containerID="f01e6c25180bc9adab08d44a682ed97fa326203f845643cd34a2d5da0f6ee012"
Nov 03 06:29:15.637804 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:15.637776    4528 scope.go:122] "RemoveContainer" containerID="9b1da406e1a27219f62f29b4483752d3a29134c9a8c081cc41a938460a818c9f"
Nov 03 06:29:34.031251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:34.031213    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"]
Nov 03 06:29:34.042321 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:34.042279    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af191ac6-mvhpp"]
Nov 03 06:29:35.748420 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:35.748389    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41dec697-134c-40b5-99b5-77846f9975f5" path="/var/lib/kubelet/pods/41dec697-134c-40b5-99b5-77846f9975f5/volumes"
Nov 03 06:29:43.692466 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:29:43.692432    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:29:49.433381 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:49.433350    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:29:49.614713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:29:49.614660    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:30:00.163885 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.163844    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:00.164348 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.164298    4528 cpu_manager.go:414] "containerMap: removing container" podUID="991ade28-f156-4851-ae9e-5924b8af073d" containerName="revoke-keys"
Nov 03 06:30:00.164348 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.164311    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991ade28-f156-4851-ae9e-5924b8af073d" containerName="revoke-keys"
Nov 03 06:30:00.164425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.164405    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="991ade28-f156-4851-ae9e-5924b8af073d" containerName="revoke-keys"
Nov 03 06:30:00.167708 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.167690    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.181881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.181841    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 06:30:00.181881 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.181859    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 06:30:00.193503 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.193482    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-6p27d\""
Nov 03 06:30:00.198608 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.198583    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:00.284286 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.284247    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.284496 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.284296    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ql8v\" (UniqueName: \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.384929 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.384892    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ql8v\" (UniqueName: \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.385121 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.385025    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.385692 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.385668    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.408026 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.407993    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ql8v\" (UniqueName: \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\") pod \"maas-api-key-cleanup-29894790-7mjcs\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.481055 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.480969    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:00.784256 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.784216    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:00.917459 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:00.917417    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" event={"ID":"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4","Type":"ContainerStarted","Data":"65b1f4f6d247686027a1ccf97118efc639c13963cd94bdee5f423645a1747bf7"}
Nov 03 06:30:01.925673 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:01.925637    4528 generic.go:356] "Generic (PLEG): container finished" podID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerID="afbc6e4367b4e23d7ab8dba81a757ac299508912526778bfd8b3cf48338b23c6" exitCode=60
Nov 03 06:30:01.926204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:01.925678    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" event={"ID":"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4","Type":"ContainerDied","Data":"afbc6e4367b4e23d7ab8dba81a757ac299508912526778bfd8b3cf48338b23c6"}
Nov 03 06:30:01.926204 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:01.926048    4528 scope.go:122] "RemoveContainer" containerID="afbc6e4367b4e23d7ab8dba81a757ac299508912526778bfd8b3cf48338b23c6"
Nov 03 06:30:02.933045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:02.932943    4528 generic.go:356] "Generic (PLEG): container finished" podID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerID="237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d" exitCode=60
Nov 03 06:30:02.933045 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:02.933011    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" event={"ID":"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4","Type":"ContainerDied","Data":"237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d"}
Nov 03 06:30:02.933606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:02.933059    4528 scope.go:122] "RemoveContainer" containerID="afbc6e4367b4e23d7ab8dba81a757ac299508912526778bfd8b3cf48338b23c6"
Nov 03 06:30:02.933606 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:02.933357    4528 scope.go:122] "RemoveContainer" containerID="237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d"
Nov 03 06:30:02.933712 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:30:02.933682    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894790-7mjcs_odh-ai-gateway-infra(fec629b8-8f9b-4d6d-bb1a-197a44cde7f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4"
Nov 03 06:30:03.941016 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:03.940986    4528 scope.go:122] "RemoveContainer" containerID="237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d"
Nov 03 06:30:03.941415 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:30:03.941237    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894790-7mjcs_odh-ai-gateway-infra(fec629b8-8f9b-4d6d-bb1a-197a44cde7f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4"
Nov 03 06:30:13.693195 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:30:13.693158    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:30:14.741171 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:14.741138    4528 scope.go:122] "RemoveContainer" containerID="237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d"
Nov 03 06:30:15.669372 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:15.669337    4528 scope.go:122] "RemoveContainer" containerID="65d86802be3a5da9413a3035f9418b155ac8492572d33f04b60ec1d7d50fcbab"
Nov 03 06:30:15.679087 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:15.679065    4528 scope.go:122] "RemoveContainer" containerID="e9253fe45a5b07d22ecc7f2df94238e59d5c21dd9ab938814de05c47d8f09d78"
Nov 03 06:30:16.017166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:16.017072    4528 generic.go:356] "Generic (PLEG): container finished" podID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerID="5593e06fd325e1d5e33574139269c47f7adafbec011059ce30c42593fdce2df4" exitCode=60
Nov 03 06:30:16.017166 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:16.017139    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" event={"ID":"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4","Type":"ContainerDied","Data":"5593e06fd325e1d5e33574139269c47f7adafbec011059ce30c42593fdce2df4"}
Nov 03 06:30:16.017705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:16.017194    4528 scope.go:122] "RemoveContainer" containerID="237dc4dc3c3bf74f150ee29afe8f4a886f2950a9237bf21cdd75f3daf54c5c6d"
Nov 03 06:30:16.017705 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:16.017592    4528 scope.go:122] "RemoveContainer" containerID="5593e06fd325e1d5e33574139269c47f7adafbec011059ce30c42593fdce2df4"
Nov 03 06:30:16.017927 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:30:16.017906    4528 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894790-7mjcs_odh-ai-gateway-infra(fec629b8-8f9b-4d6d-bb1a-197a44cde7f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4"
Nov 03 06:30:17.066304 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.066266    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:17.207841 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.207814    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:17.251731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.251551    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\" (UniqueName: \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\") pod \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") "
Nov 03 06:30:17.251731 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.251723    4528 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\") pod \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\" (UID: \"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4\") "
Nov 03 06:30:17.252523 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.252491    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca" pod "fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" (UID: "fec629b8-8f9b-4d6d-bb1a-197a44cde7f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 06:30:17.254981 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.254954    4528 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v" pod "fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" (UID: "fec629b8-8f9b-4d6d-bb1a-197a44cde7f4"). InnerVolumeSpecName "kube-api-access-4ql8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 06:30:17.353401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.353299    4528 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ql8v\" (UniqueName: \"kubernetes.io/projected/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-kube-api-access-4ql8v\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:30:17.353401 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:17.353338    4528 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4-maas-api-service-ca\") on node \"ip-10-0-132-233.ec2.internal\" DevicePath \"\""
Nov 03 06:30:18.034691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:18.034616    4528 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"
Nov 03 06:30:18.034880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:18.034612    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs" event={"ID":"fec629b8-8f9b-4d6d-bb1a-197a44cde7f4","Type":"ContainerDied","Data":"65b1f4f6d247686027a1ccf97118efc639c13963cd94bdee5f423645a1747bf7"}
Nov 03 06:30:18.034880 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:18.034748    4528 scope.go:122] "RemoveContainer" containerID="5593e06fd325e1d5e33574139269c47f7adafbec011059ce30c42593fdce2df4"
Nov 03 06:30:18.086017 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:18.085982    4528 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:18.099529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:18.099487    4528 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894790-7mjcs"]
Nov 03 06:30:19.747720 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:19.747679    4528 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" path="/var/lib/kubelet/pods/fec629b8-8f9b-4d6d-bb1a-197a44cde7f4/volumes"
Nov 03 06:30:36.588600 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:36.588521    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:30:36.775241 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:36.775211    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:30:37.148944 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.148909    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c9b79/must-gather-ptvjv"]
Nov 03 06:30:37.149389 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149375    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149391    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149409    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149442 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149417    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149509    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149537 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149521    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149626 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149616    4528 cpu_manager.go:414] "containerMap: removing container" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149625    4528 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.149698 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.149684    4528 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec629b8-8f9b-4d6d-bb1a-197a44cde7f4" containerName="cleanup"
Nov 03 06:30:37.152948 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.152929    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.170110 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.170079    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/must-gather-ptvjv"]
Nov 03 06:30:37.178298 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.178268    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9b79\"/\"kube-root-ca.crt\""
Nov 03 06:30:37.178525 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.178507    4528 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c9b79\"/\"openshift-service-ca.crt\""
Nov 03 06:30:37.178605 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.178567    4528 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c9b79\"/\"default-dockercfg-b9xm5\""
Nov 03 06:30:37.260199 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.260160    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-must-gather-output\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.260385 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.260301    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-kube-api-access-6m6kd\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.361594 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.361560    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-kube-api-access-6m6kd\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.361839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.361633    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-must-gather-output\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.361971 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.361952    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-must-gather-output\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.384582 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.384560    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m6kd\" (UniqueName: \"kubernetes.io/projected/85e4dc16-fafc-4b1d-bbfc-500d26b36bf7-kube-api-access-6m6kd\") pod \"must-gather-ptvjv\" (UID: \"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7\") " pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.468877 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.468764    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/must-gather-ptvjv"
Nov 03 06:30:37.767282 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.767254    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/must-gather-ptvjv"]
Nov 03 06:30:37.770816 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:30:37.769566    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85e4dc16_fafc_4b1d_bbfc_500d26b36bf7.slice/crio-d16db92c7ebcb0594401a83f40dd53ba49018b077e24116612f66922a4102fc6 WatchSource:0}: Error finding container d16db92c7ebcb0594401a83f40dd53ba49018b077e24116612f66922a4102fc6: Status 404 returned error can't find the container with id d16db92c7ebcb0594401a83f40dd53ba49018b077e24116612f66922a4102fc6
Nov 03 06:30:37.774251 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:37.774226    4528 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 06:30:38.158703 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:38.158663    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/must-gather-ptvjv" event={"ID":"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7","Type":"ContainerStarted","Data":"d16db92c7ebcb0594401a83f40dd53ba49018b077e24116612f66922a4102fc6"}
Nov 03 06:30:39.166363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:39.166327    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/must-gather-ptvjv" event={"ID":"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7","Type":"ContainerStarted","Data":"190cbe7c180c52dab541fff24cddb5fa7cfa7604f14cd6a37ea01ed0ba397aa1"}
Nov 03 06:30:39.166363 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:39.166368    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/must-gather-ptvjv" event={"ID":"85e4dc16-fafc-4b1d-bbfc-500d26b36bf7","Type":"ContainerStarted","Data":"21e897541f6ac54007bceb427b967f8c8068067691276d3cf489c8bbe526d1ff"}
Nov 03 06:30:39.207064 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:39.206993    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c9b79/must-gather-ptvjv" podStartSLOduration=1.452657374 podStartE2EDuration="2.206973678s" totalImagesPullingTime="754.316304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 06:30:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:30:39.204959525 +0000 UTC m=+2127.616356619" watchObservedRunningTime="2026-11-03 06:30:39.206973678 +0000 UTC m=+2127.618370757"
Nov 03 06:30:41.273425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:41.273396    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-bxj2x_6e5c002c-8b36-498b-aa36-c93aa0b3cb89/global-pull-secret-syncer/0.log"
Nov 03 06:30:42.430819 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:42.430000    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xbkcj_cac9a7a9-12dc-4699-bf59-da9a169f261d/konnectivity-agent/0.log"
Nov 03 06:30:42.538817 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:42.538649    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-132-233.ec2.internal_1b9c97e9d8c2351046139958cd95120b/haproxy/0.log"
Nov 03 06:30:43.734814 ip-10-0-132-233 kubenswrapper[4528]: E1103 06:30:43.732027    4528 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-5v8g7" containerName="extractor"
Nov 03 06:30:49.088691 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:49.088635    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-7b7bb4bc77-xjzbk_40c16b20-952a-40a3-892b-06a6dbfd0bb5/manager/0.log"
Nov 03 06:30:49.210053 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:49.207236    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-8557cbdb84-cmx7d_491fbb46-3ea5-459b-be62-ab56797e6972/manager/0.log"
Nov 03 06:30:49.323631 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:49.323594    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-vb997_b7074acb-4264-4990-b61e-f30b57821f6d/kuadrant-console-plugin/0.log"
Nov 03 06:30:49.471099 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:49.471019    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-g6r8h_dd057882-81dc-4bd5-b249-4ea76ecbe396/manager/0.log"
Nov 03 06:30:51.073576 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.073545    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/alertmanager/0.log"
Nov 03 06:30:51.128542 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.128501    4528 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"]
Nov 03 06:30:51.134343 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.134314    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.154836 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.153457    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"]
Nov 03 06:30:51.186413 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.186384    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/config-reloader/0.log"
Nov 03 06:30:51.268327 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.268291    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-lib-modules\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.268516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.268345    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-sys\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.268516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.268462    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-podres\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.268614 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.268532    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4lvs\" (UniqueName: \"kubernetes.io/projected/77c493e2-6f47-4df3-bab2-49cc1348e197-kube-api-access-z4lvs\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.268661 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.268642    4528 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-proc\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.295440 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.295404    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/kube-rbac-proxy-web/0.log"
Nov 03 06:30:51.370059 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.369968    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-podres\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370095    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4lvs\" (UniqueName: \"kubernetes.io/projected/77c493e2-6f47-4df3-bab2-49cc1348e197-kube-api-access-z4lvs\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370111    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-podres\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370137    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-proc\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370207    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-lib-modules\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370246 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370228    4528 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-sys\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370299    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-proc\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370307    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-sys\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.370499 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.370342    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/77c493e2-6f47-4df3-bab2-49cc1348e197-lib-modules\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.392351 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.392318    4528 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4lvs\" (UniqueName: \"kubernetes.io/projected/77c493e2-6f47-4df3-bab2-49cc1348e197-kube-api-access-z4lvs\") pod \"perf-node-gather-daemonset-lwt8v\" (UID: \"77c493e2-6f47-4df3-bab2-49cc1348e197\") " pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.401727 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.401704    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/kube-rbac-proxy/0.log"
Nov 03 06:30:51.454421 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.454383    4528 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:51.511179 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.511144    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/kube-rbac-proxy-metric/0.log"
Nov 03 06:30:51.622664 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.620876    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/prom-label-proxy/0.log"
Nov 03 06:30:51.732812 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.731026    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_17cc8e85-12a6-4449-9765-bed369222867/init-config-reloader/0.log"
Nov 03 06:30:51.784529 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.784494    4528 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"]
Nov 03 06:30:51.793018 ip-10-0-132-233 kubenswrapper[4528]: W1103 06:30:51.792975    4528 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod77c493e2_6f47_4df3_bab2_49cc1348e197.slice/crio-37cfea17141647e2361d2264486cd95c8c08dc0269e1f86938a753de6535c0fe WatchSource:0}: Error finding container 37cfea17141647e2361d2264486cd95c8c08dc0269e1f86938a753de6535c0fe: Status 404 returned error can't find the container with id 37cfea17141647e2361d2264486cd95c8c08dc0269e1f86938a753de6535c0fe
Nov 03 06:30:51.994787 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:51.994742    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qmc2c_1c090e4d-9e88-49f2-96fc-6f3876ab1928/kube-state-metrics/0.log"
Nov 03 06:30:52.097713 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.097679    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qmc2c_1c090e4d-9e88-49f2-96fc-6f3876ab1928/kube-rbac-proxy-main/0.log"
Nov 03 06:30:52.205754 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.205684    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-qmc2c_1c090e4d-9e88-49f2-96fc-6f3876ab1928/kube-rbac-proxy-self/0.log"
Nov 03 06:30:52.423500 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.423457    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v" event={"ID":"77c493e2-6f47-4df3-bab2-49cc1348e197","Type":"ContainerStarted","Data":"6da2bb86c437605750ad15acfd44d6a99e282aaf4de910de9c481d6ba86bcf40"}
Nov 03 06:30:52.423862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.423528    4528 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v" event={"ID":"77c493e2-6f47-4df3-bab2-49cc1348e197","Type":"ContainerStarted","Data":"37cfea17141647e2361d2264486cd95c8c08dc0269e1f86938a753de6535c0fe"}
Nov 03 06:30:52.423862 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.423579    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:52.465332 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:52.465212    4528 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v" podStartSLOduration=1.465191023 podStartE2EDuration="1.465191023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 06:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 06:30:52.463439467 +0000 UTC m=+2140.874836546" watchObservedRunningTime="2026-11-03 06:30:52.465191023 +0000 UTC m=+2140.876588119"
Nov 03 06:30:53.232946 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.232917    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k96cf_8f65d210-43a3-41de-a608-1bf820be5620/node-exporter/0.log"
Nov 03 06:30:53.339216 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.339170    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k96cf_8f65d210-43a3-41de-a608-1bf820be5620/kube-rbac-proxy/0.log"
Nov 03 06:30:53.444080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.444025    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k96cf_8f65d210-43a3-41de-a608-1bf820be5620/init-textfile/0.log"
Nov 03 06:30:53.555706 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.555673    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c2c5q_1e398089-1045-4b35-bcad-f9649b94a78c/kube-rbac-proxy-main/0.log"
Nov 03 06:30:53.660657 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.660626    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c2c5q_1e398089-1045-4b35-bcad-f9649b94a78c/kube-rbac-proxy-self/0.log"
Nov 03 06:30:53.765518 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:53.765478    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-c2c5q_1e398089-1045-4b35-bcad-f9649b94a78c/openshift-state-metrics/0.log"
Nov 03 06:30:54.993658 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:54.993614    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bd556c7-rpgfk_63835035-86c3-4afd-ac62-7eaa1165e159/telemeter-client/0.log"
Nov 03 06:30:55.102750 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:55.102718    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bd556c7-rpgfk_63835035-86c3-4afd-ac62-7eaa1165e159/reload/0.log"
Nov 03 06:30:55.210590 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:55.210553    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6f7bd556c7-rpgfk_63835035-86c3-4afd-ac62-7eaa1165e159/kube-rbac-proxy/0.log"
Nov 03 06:30:58.446516 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:58.446485    4528 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c9b79/perf-node-gather-daemonset-lwt8v"
Nov 03 06:30:59.266080 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:30:59.266047    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-74b667d744-r9f6w_2a33c2da-9397-440c-aaef-78808c182b6c/console/0.log"
Nov 03 06:31:03.413027 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:03.412982    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8jsb_7c828422-e7da-4ee5-b748-8e8827dded89/dns/0.log"
Nov 03 06:31:03.519425 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:03.519395    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p8jsb_7c828422-e7da-4ee5-b748-8e8827dded89/kube-rbac-proxy/0.log"
Nov 03 06:31:03.861737 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:03.861693    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hvprb_81d514bd-df1e-428c-ab18-66f841c88928/dns-node-resolver/0.log"
Nov 03 06:31:04.271743 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:04.271659    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485wjs6f_aba44b32-7a71-462f-a683-7d6e39f140ac/istio-proxy/0.log"
Nov 03 06:31:05.052411 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.052372    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7b94465567-lmkz8_9f0c653e-c147-4c5d-8c46-9e9db71a9feb/registry/0.log"
Nov 03 06:31:05.167734 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.167702    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-bws8t_23536828-f0dc-4f42-8f23-988d8de6a262/node-ca/0.log"
Nov 03 06:31:05.508980 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.508895    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-8mbmr_11852422-e9b2-4cc4-a315-644dd38888f9/istio-proxy/0.log"
Nov 03 06:31:05.636308 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.636275    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/0.log"
Nov 03 06:31:05.638018 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.637992    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/1.log"
Nov 03 06:31:05.857536 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.857496    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-q2zk7_138857e2-d465-4298-a8af-7ebfbab95b86/istio-proxy/0.log"
Nov 03 06:31:05.977517 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:05.977480    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-zw8r9_bdae4c2a-d4ca-4749-8c88-a62fbe20a3bb/istio-proxy/0.log"
Nov 03 06:31:06.033890 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:06.033862    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-75b555675f-7ltk4_63f8190f-d2db-4674-b2dc-7894853d97ec/istio-proxy/0.log"
Nov 03 06:31:06.091322 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:06.091287    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9m2mq_1c767011-9f84-4a57-9d95-727814cc25e4/payload-pre-processing/0.log"
Nov 03 06:31:06.147241 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:06.147155    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-fggx5_ddfcd2d1-6719-44f9-bff2-b0d5c937b50b/payload-processing/0.log"
Nov 03 06:31:07.164887 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:07.164858    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485wjs6f_aba44b32-7a71-462f-a683-7d6e39f140ac/istio-proxy/0.log"
Nov 03 06:31:08.714114 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:08.714081    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-8mbmr_11852422-e9b2-4cc4-a315-644dd38888f9/istio-proxy/0.log"
Nov 03 06:31:08.805193 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:08.805161    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-58c98c4485wjs6f_aba44b32-7a71-462f-a683-7d6e39f140ac/istio-proxy/0.log"
Nov 03 06:31:08.938132 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:08.938097    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/0.log"
Nov 03 06:31:08.940775 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:08.940753    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-c4nqk_e8c4a3ba-5a15-40ac-823e-795c43744b3c/istio-proxy/1.log"
Nov 03 06:31:09.428839 ip-10-0-132-233 kubenswrapper[4528]: I1103 06:31:09.428785    4528 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-778b7b5c47-q2zk7_138857e2-d465-4298-a8af-7ebfbab95b86/istio-proxy/0.log"